Friday, 19 September 2014

how to transforming pseudo xml into flat structure?





I'm trying to parse file that looks like xml but is not. Actualy it is readable version of CRD transformed from ASN1 format. It looks like this:



<PIN rowNum="1">
<CgPa tag="3100.2.960.51" value="1">
<data tag="3100.2.962.56" name="cgPASubscriberIdentifier" value="50212000000089804"/>
<data tag="3100.2.962.60" name="cgPaRoaming" value="1"/>
</CgPa>
<AAA_Common tag="3100.2.960.1" value="1">
<data tag="3100.2.962.12" name="sigSleeId" value="watbf102"/>
<data tag="3100.2.962.34" name="scpAddress" value="48602888950"/>
</AAA_Common>
<evt tag="3100.2.134.28" name="unsupported" value="0"/>
<data tag="3100.2.112.1" name="eventDateTime" value="07/05/2014 19:45:18"/>
<data tag="3100.2.137.4" name="inTriggeringKey" value="0048662221827"/>
<evt tag="3100.2.137.5" name="typeINTriggeringKey" value="1"/>
<CustomerDomain tag="3100.2.134.1" value="1">
<data tag="3100.2.133.1" name="ordinaryClientId" value="50212000000089804"/>
<data tag="3100.2.105.1" name="customerServiceName" value="SO_TT_Roam_Voice"/>
<AccountDomain tag="3100.2.134.3" value="1">
<data tag="3100.2.104.4" name="accountIdentifier" value="50212000000089804"/>
<data tag="3100.2.100.1" name="subscriberType" value="1"/>
<evt tag="3100.2.139.3" name="unsupported" value="0"/>
<TariffDomain tag="3100.2.134.11" value="1">
<data tag="3100.2.106.10" name="tariffPlanNameVersion" value="TT_VOI_R_1_PL_1A_0_RoamB - 2_TCA"/>
</TariffDomain>
<TariffDomain tag="3100.2.134.11" value="1">
<data tag="3100.2.106.10" name="tariffPlanNameVersion" value="TT_VOI_R_1_PL_1A_0_Main - 2_TCA"/>
<data tag="3100.2.106.1" name="tariffPlanName" value="TT_VOI_R_1_PL_1A_0_Main"/>
<evt tag="3100.2.139.9" name="tariffCost" value="1013"/>
<evt tag="3100.2.139.10" name="tariffCostVat" value="1013"/>
<evt tag="3100.2.140.7" name="eventQuantityPerTariff1" value="614"/>
<evt tag="3100.2.142.11" name="usedQuantityPerTariff1" value="614"/>
</TariffDomain>
<evt tag="3100.2.134.29" name="unsupported" value="1"/>
<data tag="3100.2.124.45" name="unsupported" value="07/05/2014 19:45:18"/>
<evt tag="3100.2.139.35" name="unsupported" value="495"/>
<data tag="3100.2.24.11" name="unsupported" value="84490"/>
<evt tag="3100.2.134.30" name="unsupported" value="1"/>
</AccountDomain>
</CustomerDomain>
</PIN>


The main tag for each record is PIN, but sub-tags can appear in random order or don't appear at all. Typical solution for xml cases in pig is to use piggybank function XMLLoader. But it assumes that order of tags is constant. Otherwise we are unnable to put it into schema. The only solution I see it to REGEXP each line and take name and value and use map[]. But what about tags that appear more that ones like TariffDomain in my example? How do I deal with it?


Regards

Pawel


No comments:

Post a Comment