SunQuest
           Database Help
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Codewalkers ForumsOther TechnologiesDatabase Help

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Codewalkers Forums Sponsor:
Stay one step ahead of the competition. Evaluate and give feedback on some of the hottest web development tools on the market today. Make your opinion heard! Click Here
  #1  
Old July 29th, 2003, 08:29 PM
mugane mugane is offline
Codewalkers Newbie (0 - 499 posts)
 
Join Date: Apr 2007
Location: Charlottesville, VA USA
Posts: 425 mugane User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 2
Send a message via AIM to mugane Send a message via Yahoo to mugane
Baffled by incomplete insert - MySQL

Input: 2308 text files, each containing 250 lines, comma-delimited fields, all ~137Kb.

Function: The script reads in the text from the files, parses it into arrays of data that correspond with the MySQL table, and then inserts them into the table one record at a time.

Output: none

It works perfectly for the first ~6,400,000 records (843 input files), and then it stops somewhere in the middle of the 844th file. The size of the database (.MYD file) is 407Mb at this point, and there are at least 700Mb free on the drive.
I know that my query is not erroneous (it inserts the first 6 million records).
I have also executed it without running the query portion, and it completes all 2308 files) - so the input and parsing are not erroneous either.

My table information (SHOW TABLE STATUS) is as follows:
Code:
type: MyISAM
Row Format: Dynamic
Rows: 6415767
Avg. Row Length: 64
Data Length: 417410720
Max Data Length: 4294967295
Index Length: 79093760
Data Free: 0
etc etc


Any ideas? Why does it say that 'Data free: 0'?

Thanks for any tips,

Peter

Reply With Quote
  #2  
Old July 29th, 2003, 08:49 PM
Blindeddie Blindeddie is offline
Codewalkers Regular (2000 - 2499 posts)
 
Join Date: Apr 2007
Location: NJ - USA
Posts: 2,152 Blindeddie User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 4
RE: Baffled by incomplete insert - MySQL

check this out...

http://www.mysql.com/doc/en/SHOW_TABLE_STATUS.html

Reply With Quote
  #3  
Old July 29th, 2003, 08:56 PM
mugane mugane is offline
Codewalkers Newbie (0 - 499 posts)
 
Join Date: Apr 2007
Location: Charlottesville, VA USA
Posts: 425 mugane User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 2
Send a message via AIM to mugane Send a message via Yahoo to mugane
RE: Baffled by incomplete insert - MySQL

Yeah, that's how I got the info I posted above... I'm not sure I follow...

Reply With Quote
  #4  
Old July 29th, 2003, 09:05 PM
Blindeddie Blindeddie is offline
Codewalkers Regular (2000 - 2499 posts)
 
Join Date: Apr 2007
Location: NJ - USA
Posts: 2,152 Blindeddie User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 4
RE: Baffled by incomplete insert - MySQL

scroll down for the explaination of each of the results. (page is formatted weird) It looks like you might be reaching the data limit on the table. at least that is how I read it.

Reply With Quote
  #5  
Old July 29th, 2003, 09:17 PM
mugane mugane is offline
Codewalkers Newbie (0 - 499 posts)
 
Join Date: Apr 2007
Location: Charlottesville, VA USA
Posts: 425 mugane User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 2
Send a message via AIM to mugane Send a message via Yahoo to mugane
RE: Baffled by incomplete insert - MySQL

Hm. I see - sorry about that, I did not scroll when the page came up with all that whitespace...

Well, you may be right, do you think I should increase the maximum size of the database or make the rows fixed (as opposed to dynamic)?

Thanks,

Peter

Reply With Quote
  #6  
Old July 29th, 2003, 09:27 PM
mugane mugane is offline
Codewalkers Newbie (0 - 499 posts)
 
Join Date: Apr 2007
Location: Charlottesville, VA USA
Posts: 425 mugane User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 2
Send a message via AIM to mugane Send a message via Yahoo to mugane
RE: Baffled by incomplete insert - MySQL

Hm. I can't seem to figure out how to increase this grr max_data_length ... do you know how it's done? where can I find 'max_rows'? It's not in my my.ini or my.cnf files... is it table-specific? Ideally I would like to increase it on the table that already has data in it because even my great script takes a few hours to insert all that data...

Thanks again,

P

Reply With Quote
  #7  
Old July 29th, 2003, 09:34 PM
mugane mugane is offline
Codewalkers Newbie (0 - 499 posts)
 
Join Date: Apr 2007
Location: Charlottesville, VA USA
Posts: 425 mugane User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 2
Send a message via AIM to mugane Send a message via Yahoo to mugane
RE: Baffled by incomplete insert - MySQL

Ok the manual says it's done at table creation, and can be changed using ALTER... but no example was given...

Reply With Quote
  #8  
Old July 29th, 2003, 09:42 PM
mugane mugane is offline
Codewalkers Newbie (0 - 499 posts)
 
Join Date: Apr 2007
Location: Charlottesville, VA USA
Posts: 425 mugane User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 2
Send a message via AIM to mugane Send a message via Yahoo to mugane
RE: Baffled by incomplete insert - MySQL

Ok I have it.

ALTER TABLE tablename MAX_ROWS = 1000000000;

Let's see what happens (I probably won't know until tomorrow)

Thanks, I've marked as resolved until I get stuck again...

Peter

Reply With Quote
  #9  
Old July 29th, 2003, 10:08 PM
bakertrg's Avatar
bakertrg bakertrg is offline
Contributing User
Codewalkers Regular (2000 - 2499 posts)
 
Join Date: Apr 2007
Location: Scottsdale AZ, US
Posts: 2,253 bakertrg User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 2 h 48 m 45 sec
Reputation Power: 4
Send a message via Yahoo to bakertrg
RE: Baffled by incomplete insert - MySQL

Just a thought but I had a DB where I intially set the autoincrement id field to a value that seemed large enough at the time but years later was maxed out and it took me a while to realize what the problem was.

Reply With Quote
  #10  
Old July 29th, 2003, 10:20 PM
zombie zombie is offline
Codewalkers Intermediate (1500 - 1999 posts)
 
Join Date: Apr 2007
Location: serbia
Posts: 1,876 zombie User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 3
RE: Baffled by incomplete insert - MySQL

why don't you use mysql_error() to see the error you are getting. use it like so:

mysql_query($blah) or die(mysql_error());


than we wouldn't feel like we are wondering in the dark...

Reply With Quote
  #11  
Old July 30th, 2003, 05:56 PM
mugane mugane is offline
Codewalkers Newbie (0 - 499 posts)
 
Join Date: Apr 2007
Location: Charlottesville, VA USA
Posts: 425 mugane User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 2
Send a message via AIM to mugane Send a message via Yahoo to mugane
RE: Baffled by incomplete insert - MySQL

Lo and behold it works.

Thanks all for the input on this one!

P

Reply With Quote
Reply

Viewing: Codewalkers ForumsOther TechnologiesDatabase Help > Baffled by incomplete insert - MySQL


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 5 hosted by Hostway