XML : Discuss better way update huge data XML files or Mysql InnoDB

I make big web service with huge amount updating data. Millions of items updated every 30 minutes with data from another servers. No need search in this data just get data by unique number ID.

I create InnoDB table with data table. I parse coming data and put it in table rows. On stream coming data 10Mbit per sec from servers. HDD usage was 100%.

Then i try just save data in text xml files on server in different folders

  dir/1.xml  dir/2.xml  dir2/3.xml  dir2/4.xml    

because PHP scripts which get data from servers cannot save files in some folders because write operation locked(delayed) access for other scripts which working at same time. (Maybe i need create folder for each data file?) Result is better i get around %5 HDD usage at same incoming data stream 10Mbit/s.

So this data i use for show on web page via PHP script.

Maybe someone know better way?

No comments:

Post a Comment