18 June 2009

Convert single row INSERTs to multi-row INSERT

Ronald's post regarding multi-row INSERTs with MySQL reminded me of a quick utility I wrote about a year ago. I wanted to test for myself how much performance can be gained over singleton INSERTs. The perl script below just takes the output from mysqldump and produces a file for each table, with the file containing the table DDL and the associated INSERT statements. Thought others may find it useful so here it is

-------- cut here ---------

#!/usr/bin/perl -W
#
##
##------------------------------------------------------------------------
## multi_inserts.pl
## ------------
##
## Take singular INSERT statements and create multi-row INSERT statements.
##------------------------------------------------------------------------
##
# Modification History
# --------------------
# C Sylvester, 3 July 2008, Initial release

use strict;

my $VALUES_PER_INSERT = 50;
my $MAX_ROW_LENGTH = $VALUES_PER_INSERT * 1000;

my $initial_comments = "";
my $table = "";
my $cnt = 0;
my $multi_values = "";
my $multi_row_length = 0;
my $total_row_count = 0;

my $t = "";
my $v = "";

open( PF, $ARGV[0] ) or die "\"$ARGV[0]\": $!\nUsage $0 \n";

LINE: while () {

if ( /^INSERT INTO `(\w+)` VALUES (.*);/ ) {
$t = $1;
$v = $2;
if ( $table eq "" ) {
$table = lc ($t).".sql";
open (DDL, '>>', $table) or die "Cannot open file for appending\n$!\n";
}

$multi_values .= $v . ",";
$multi_row_length += length($v);
if ( ++$cnt == $VALUES_PER_INSERT || $multi_row_length >= $MAX_ROW_SIZE) {
$multi_values =~ s/,$/\;/;
print DDL "INSERT INTO ", $t, " VALUES ", $multi_values, "\n";
$total_row_count += $cnt;
$cnt = 0;
$multi_values = "";
$multi_row_length = 0;
}

next LINE;
}

if ( $cnt != 0 ) {
$multi_values =~ s/,$/\;/;
print DDL "INSERT INTO ", $t, " VALUES ", $multi_values, "\n";
$total_row_count += $cnt;
$cnt = 0;
$multi_values = "";
$multi_row_length = 0;

}

if (/-- MySQL dump .*/ ) {
do  {
$initial_comments .= $_;
$_ = ;
} until ( /-- Table structure for table .*/ )
}

if ( /-- Table structure for table `(\w+)`/ ) {
$table = lc ($+).".sql";

close(DDL);
open (DDL, '+>', $table) or die "Cannot open file for writing\n$!\n";

print DDL $initial_comments, "\n";

print STDOUT "Printing to ", $table, "\n";
}

print DDL if fileno DDL;
}

if ( $cnt != 0 ) {
$multi_values =~ s/,$/\;/;
$total_row_count += $cnt;
print DDL "INSERT INTO ", $t, " VALUES ", $multi_values, "\n";
}

print STDOUT "Table \"", $t, "\" = ", $total_row_count;

close(DDL);
close(PF);
exit;


-------- cut here ---------

Part of my motivation for doing this was to learn PERL a little better so I'm sure this could be improved :-)

No comments:

Post a Comment