Analyse tables with unknown structure and fault tolerance











up vote
1
down vote

favorite












I want to analyse tables with similar data, that are structured differently and where the headers also may be slightly diverse.



For collecting all the data from the tables summing them up I face several problems.



Step 1: I look for the header keywords. Searching for if "cars==cars" is not possible, because the header may appear as "car", "Car" or "Cars". There is also the possibilty that there is a spelling mistake in the word. So iterating through all possibilites can also result in false.
When I search for solutions to this problem I found out about the fuzzy logic, but I would be thankful about other approaches.



Step 2: I found the desired keyword in the table, but how do I know where the related data is placed? It can be below it, but also right cell next to it. Are there approaches to get information about the general structure of the table?










share|improve this question




























    up vote
    1
    down vote

    favorite












    I want to analyse tables with similar data, that are structured differently and where the headers also may be slightly diverse.



    For collecting all the data from the tables summing them up I face several problems.



    Step 1: I look for the header keywords. Searching for if "cars==cars" is not possible, because the header may appear as "car", "Car" or "Cars". There is also the possibilty that there is a spelling mistake in the word. So iterating through all possibilites can also result in false.
    When I search for solutions to this problem I found out about the fuzzy logic, but I would be thankful about other approaches.



    Step 2: I found the desired keyword in the table, but how do I know where the related data is placed? It can be below it, but also right cell next to it. Are there approaches to get information about the general structure of the table?










    share|improve this question


























      up vote
      1
      down vote

      favorite









      up vote
      1
      down vote

      favorite











      I want to analyse tables with similar data, that are structured differently and where the headers also may be slightly diverse.



      For collecting all the data from the tables summing them up I face several problems.



      Step 1: I look for the header keywords. Searching for if "cars==cars" is not possible, because the header may appear as "car", "Car" or "Cars". There is also the possibilty that there is a spelling mistake in the word. So iterating through all possibilites can also result in false.
      When I search for solutions to this problem I found out about the fuzzy logic, but I would be thankful about other approaches.



      Step 2: I found the desired keyword in the table, but how do I know where the related data is placed? It can be below it, but also right cell next to it. Are there approaches to get information about the general structure of the table?










      share|improve this question















      I want to analyse tables with similar data, that are structured differently and where the headers also may be slightly diverse.



      For collecting all the data from the tables summing them up I face several problems.



      Step 1: I look for the header keywords. Searching for if "cars==cars" is not possible, because the header may appear as "car", "Car" or "Cars". There is also the possibilty that there is a spelling mistake in the word. So iterating through all possibilites can also result in false.
      When I search for solutions to this problem I found out about the fuzzy logic, but I would be thankful about other approaches.



      Step 2: I found the desired keyword in the table, but how do I know where the related data is placed? It can be below it, but also right cell next to it. Are there approaches to get information about the general structure of the table?







      algorithm data-analysis tabular






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 8 at 16:43









      Brian Tompsett - 汤莱恩

      4,153133699




      4,153133699










      asked Nov 8 at 9:34









      thohemp

      63




      63
























          1 Answer
          1






          active

          oldest

          votes

















          up vote
          0
          down vote













          Step a (part 1) - naive implementation would be dictionary distance (as you want to handle typos)



          Step a (part 2) - use synonym database / thesaurus to find similarly named columns



          Step b (part 1) - data is aligned the same way the headers are - so if headers are aligned vertically, then data will be as well



          Step b (part 2) - similar data will have the similar data type (raw string, number, zip-code), by checking to the right and to downwards you can detect which is the real direction.






          share|improve this answer





















            Your Answer






            StackExchange.ifUsing("editor", function () {
            StackExchange.using("externalEditor", function () {
            StackExchange.using("snippets", function () {
            StackExchange.snippets.init();
            });
            });
            }, "code-snippets");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "1"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














             

            draft saved


            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53204936%2fanalyse-tables-with-unknown-structure-and-fault-tolerance%23new-answer', 'question_page');
            }
            );

            Post as a guest
































            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes








            up vote
            0
            down vote













            Step a (part 1) - naive implementation would be dictionary distance (as you want to handle typos)



            Step a (part 2) - use synonym database / thesaurus to find similarly named columns



            Step b (part 1) - data is aligned the same way the headers are - so if headers are aligned vertically, then data will be as well



            Step b (part 2) - similar data will have the similar data type (raw string, number, zip-code), by checking to the right and to downwards you can detect which is the real direction.






            share|improve this answer

























              up vote
              0
              down vote













              Step a (part 1) - naive implementation would be dictionary distance (as you want to handle typos)



              Step a (part 2) - use synonym database / thesaurus to find similarly named columns



              Step b (part 1) - data is aligned the same way the headers are - so if headers are aligned vertically, then data will be as well



              Step b (part 2) - similar data will have the similar data type (raw string, number, zip-code), by checking to the right and to downwards you can detect which is the real direction.






              share|improve this answer























                up vote
                0
                down vote










                up vote
                0
                down vote









                Step a (part 1) - naive implementation would be dictionary distance (as you want to handle typos)



                Step a (part 2) - use synonym database / thesaurus to find similarly named columns



                Step b (part 1) - data is aligned the same way the headers are - so if headers are aligned vertically, then data will be as well



                Step b (part 2) - similar data will have the similar data type (raw string, number, zip-code), by checking to the right and to downwards you can detect which is the real direction.






                share|improve this answer












                Step a (part 1) - naive implementation would be dictionary distance (as you want to handle typos)



                Step a (part 2) - use synonym database / thesaurus to find similarly named columns



                Step b (part 1) - data is aligned the same way the headers are - so if headers are aligned vertically, then data will be as well



                Step b (part 2) - similar data will have the similar data type (raw string, number, zip-code), by checking to the right and to downwards you can detect which is the real direction.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Nov 8 at 17:03









                Adam Kotwasinski

                1,818521




                1,818521






























                     

                    draft saved


                    draft discarded



















































                     


                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53204936%2fanalyse-tables-with-unknown-structure-and-fault-tolerance%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest




















































































                    Popular posts from this blog

                    鏡平學校

                    ꓛꓣだゔៀៅຸ໢ທຮ໕໒ ,ໂ'໥໓າ໼ឨឲ៵៭ៈゎゔit''䖳𥁄卿' ☨₤₨こゎもょの;ꜹꟚꞖꞵꟅꞛေၦေɯ,ɨɡ𛃵𛁹ޝ޳ޠ޾,ޤޒޯ޾𫝒𫠁သ𛅤チョ'サノބޘދ𛁐ᶿᶇᶀᶋᶠ㨑㽹⻮ꧬ꧹؍۩وَؠ㇕㇃㇪ ㇦㇋㇋ṜẰᵡᴠ 軌ᵕ搜۳ٰޗޮ޷ސޯ𫖾𫅀ल, ꙭ꙰ꚅꙁꚊꞻꝔ꟠Ꝭㄤﺟޱސꧨꧼ꧴ꧯꧽ꧲ꧯ'⽹⽭⾁⿞⼳⽋២៩ញណើꩯꩤ꩸ꩮᶻᶺᶧᶂ𫳲𫪭𬸄𫵰𬖩𬫣𬊉ၲ𛅬㕦䬺𫝌𫝼,,𫟖𫞽ហៅ஫㆔ాఆఅꙒꚞꙍ,Ꙟ꙱エ ,ポテ,フࢰࢯ𫟠𫞶 𫝤𫟠ﺕﹱﻜﻣ𪵕𪭸𪻆𪾩𫔷ġ,ŧآꞪ꟥,ꞔꝻ♚☹⛵𛀌ꬷꭞȄƁƪƬșƦǙǗdžƝǯǧⱦⱰꓕꓢႋ神 ဴ၀க௭எ௫ឫោ ' េㇷㇴㇼ神ㇸㇲㇽㇴㇼㇻㇸ'ㇸㇿㇸㇹㇰㆣꓚꓤ₡₧ ㄨㄟ㄂ㄖㄎ໗ツڒذ₶।ऩछएोञयूटक़कयँृी,冬'𛅢𛅥ㇱㇵㇶ𥄥𦒽𠣧𠊓𧢖𥞘𩔋цѰㄠſtʯʭɿʆʗʍʩɷɛ,əʏダヵㄐㄘR{gỚṖḺờṠṫảḙḭᴮᵏᴘᵀᵷᵕᴜᴏᵾq﮲ﲿﴽﭙ軌ﰬﶚﶧ﫲Ҝжюїкӈㇴffצּ﬘﭅﬈軌'ffistfflſtffतभफɳɰʊɲʎ𛁱𛁖𛁮𛀉 𛂯𛀞నఋŀŲ 𫟲𫠖𫞺ຆຆ ໹້໕໗ๆทԊꧢꧠ꧰ꓱ⿝⼑ŎḬẃẖỐẅ ,ờỰỈỗﮊDžȩꭏꭎꬻ꭮ꬿꭖꭥꭅ㇭神 ⾈ꓵꓑ⺄㄄ㄪㄙㄅㄇstA۵䞽ॶ𫞑𫝄㇉㇇゜軌𩜛𩳠Jﻺ‚Üမ႕ႌႊၐၸဓၞၞၡ៸wyvtᶎᶪᶹစဎ꣡꣰꣢꣤ٗ؋لㇳㇾㇻㇱ㆐㆔,,㆟Ⱶヤマފ޼ޝަݿݞݠݷݐ',ݘ,ݪݙݵ𬝉𬜁𫝨𫞘くせぉて¼óû×ó£…𛅑הㄙくԗԀ5606神45,神796'𪤻𫞧ꓐ㄁ㄘɥɺꓵꓲ3''7034׉ⱦⱠˆ“𫝋ȍ,ꩲ軌꩷ꩶꩧꩫఞ۔فڱێظペサ神ナᴦᵑ47 9238їﻂ䐊䔉㠸﬎ffiﬣ,לּᴷᴦᵛᵽ,ᴨᵤ ᵸᵥᴗᵈꚏꚉꚟ⻆rtǟƴ𬎎

                    Why https connections are so slow when debugging (stepping over) in Java?