Exporting lot's of Oracle nested data to XML file in .NET



I need to export large quantities of data to XML file from Oracle database. Size of exported files may vary from few MB to few hundred MB (few MB most of the time). The structure of exported object is nested on few levels (tree of objects). Each nesting level (tree level) is a separate table. Exported data are filtered by few properties on each level. I am using .NET framework to execute the whole process.


The structure looks something like that (names modified due to NDA):



Packages -* MajorVersions -* MinorVersions -* Features (lots)
/ | \
*Excludes *Includes *Rules (lots)
/ | \
*A B *C


I search Rules that might be filtered by package or version or feature or all of them. And I want to export those Rules with appropriate parent information, so XML looks like the structure.


What kind of solution is appropriate here (efficient, with maintainable code)? Actually we are rewriting old system that executed those exports very slowly. It loaded each level to the memory, loading every child by id, so it might have performed for example about 20k requests to database to prepare an export. Then after recreating the whole object structure in the memory it serialized it to the XML. After that it performed some XSLT transformation. So export of 40MB file with about 20k Rules lasted about 30 minutes. We want to make it much more efficient and I am considering few options:



  1. Create Sql query with multiple JOINS from the top to the bottom of the tree to retrieve the most meaningful data with one select. Use ADO.NET to sequentially retrieve data with SqlDataReader and write them sequentially to the file with XMLWriter in a loop (fetch row, write XML entry). Meanwhile fetch less meaningful data (like Excludes or Includes) or load some rows at once into the memory if there are not many of them. Low memory usage seems to be an advantage here.

  2. Use ORM (NHibernate) to filter objects, load them to memory and later save to XML. It will probably be easier to implement and maintain. But it seems that ORMs are not designed to handle large data quantity cases.

  3. Use Oracle XML creation mechanism. I don't know whether it is possible to execute it with .NET and whether it's efficient. The database is located on a remote server.


Do you have any suggestions?


No comments:

Post a Comment