How to apply scipy.signal.filtfilt() on incomplete data











up vote
0
down vote

favorite












I want to plot incomplete data (some values are None). In addition I want to apply a butter function on the dataset and show both graphs, incomplete and smoothened. The filter function seems to not work with incomplete data.



Data File: data.csv



import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from scipy import signal

data = np.genfromtxt('data.csv', delimiter = ',')
df = pd.DataFrame(data)
df.set_index(0, inplace = True)

b, a = signal.butter(5, 0.1)
y = signal.filtfilt(b,a, df[1].values)
df2 = pd.DataFrame(y, index=df.index)

df.plot()
df2.plot()

plt.show()


enter image description hereenter image description here










share|improve this question


























    up vote
    0
    down vote

    favorite












    I want to plot incomplete data (some values are None). In addition I want to apply a butter function on the dataset and show both graphs, incomplete and smoothened. The filter function seems to not work with incomplete data.



    Data File: data.csv



    import matplotlib.pyplot as plt
    import pandas as pd
    import numpy as np
    from scipy import signal

    data = np.genfromtxt('data.csv', delimiter = ',')
    df = pd.DataFrame(data)
    df.set_index(0, inplace = True)

    b, a = signal.butter(5, 0.1)
    y = signal.filtfilt(b,a, df[1].values)
    df2 = pd.DataFrame(y, index=df.index)

    df.plot()
    df2.plot()

    plt.show()


    enter image description hereenter image description here










    share|improve this question
























      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      I want to plot incomplete data (some values are None). In addition I want to apply a butter function on the dataset and show both graphs, incomplete and smoothened. The filter function seems to not work with incomplete data.



      Data File: data.csv



      import matplotlib.pyplot as plt
      import pandas as pd
      import numpy as np
      from scipy import signal

      data = np.genfromtxt('data.csv', delimiter = ',')
      df = pd.DataFrame(data)
      df.set_index(0, inplace = True)

      b, a = signal.butter(5, 0.1)
      y = signal.filtfilt(b,a, df[1].values)
      df2 = pd.DataFrame(y, index=df.index)

      df.plot()
      df2.plot()

      plt.show()


      enter image description hereenter image description here










      share|improve this question













      I want to plot incomplete data (some values are None). In addition I want to apply a butter function on the dataset and show both graphs, incomplete and smoothened. The filter function seems to not work with incomplete data.



      Data File: data.csv



      import matplotlib.pyplot as plt
      import pandas as pd
      import numpy as np
      from scipy import signal

      data = np.genfromtxt('data.csv', delimiter = ',')
      df = pd.DataFrame(data)
      df.set_index(0, inplace = True)

      b, a = signal.butter(5, 0.1)
      y = signal.filtfilt(b,a, df[1].values)
      df2 = pd.DataFrame(y, index=df.index)

      df.plot()
      df2.plot()

      plt.show()


      enter image description hereenter image description here







      python pandas numpy matplotlib scipy






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 12 at 12:16









      Artur Müller Romanov

      367212




      367212
























          1 Answer
          1






          active

          oldest

          votes

















          up vote
          1
          down vote



          accepted










          The documentation page does not mention anything related to NaN. You may have to first remove the NaN from your list of values. Here is a way to do it using Numpy isnan function:



          y = signal.filtfilt(b, a, df[1].values[~np.isnan(df[1].values)])
          df2 = pd.DataFrame(y, index=df.index[~np.isnan(df[1].values)])





          share|improve this answer



















          • 1




            That might be a reasonable solution. But the filter functions in scipy.signal assume the data is sampled at regular intervals. If you discard the nan values, the sample periods of the remaining data are no longer consistent, and that may invalidate the filter results.
            – Warren Weckesser
            Nov 12 at 14:00










          • In this case one would have to first manually resample the data at a lower frequency to match the removal of NaN values ?
            – Patol75
            Nov 12 at 14:39











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53262014%2fhow-to-apply-scipy-signal-filtfilt-on-incomplete-data%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          up vote
          1
          down vote



          accepted










          The documentation page does not mention anything related to NaN. You may have to first remove the NaN from your list of values. Here is a way to do it using Numpy isnan function:



          y = signal.filtfilt(b, a, df[1].values[~np.isnan(df[1].values)])
          df2 = pd.DataFrame(y, index=df.index[~np.isnan(df[1].values)])





          share|improve this answer



















          • 1




            That might be a reasonable solution. But the filter functions in scipy.signal assume the data is sampled at regular intervals. If you discard the nan values, the sample periods of the remaining data are no longer consistent, and that may invalidate the filter results.
            – Warren Weckesser
            Nov 12 at 14:00










          • In this case one would have to first manually resample the data at a lower frequency to match the removal of NaN values ?
            – Patol75
            Nov 12 at 14:39















          up vote
          1
          down vote



          accepted










          The documentation page does not mention anything related to NaN. You may have to first remove the NaN from your list of values. Here is a way to do it using Numpy isnan function:



          y = signal.filtfilt(b, a, df[1].values[~np.isnan(df[1].values)])
          df2 = pd.DataFrame(y, index=df.index[~np.isnan(df[1].values)])





          share|improve this answer



















          • 1




            That might be a reasonable solution. But the filter functions in scipy.signal assume the data is sampled at regular intervals. If you discard the nan values, the sample periods of the remaining data are no longer consistent, and that may invalidate the filter results.
            – Warren Weckesser
            Nov 12 at 14:00










          • In this case one would have to first manually resample the data at a lower frequency to match the removal of NaN values ?
            – Patol75
            Nov 12 at 14:39













          up vote
          1
          down vote



          accepted







          up vote
          1
          down vote



          accepted






          The documentation page does not mention anything related to NaN. You may have to first remove the NaN from your list of values. Here is a way to do it using Numpy isnan function:



          y = signal.filtfilt(b, a, df[1].values[~np.isnan(df[1].values)])
          df2 = pd.DataFrame(y, index=df.index[~np.isnan(df[1].values)])





          share|improve this answer














          The documentation page does not mention anything related to NaN. You may have to first remove the NaN from your list of values. Here is a way to do it using Numpy isnan function:



          y = signal.filtfilt(b, a, df[1].values[~np.isnan(df[1].values)])
          df2 = pd.DataFrame(y, index=df.index[~np.isnan(df[1].values)])






          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Nov 12 at 12:54

























          answered Nov 12 at 12:33









          Patol75

          6136




          6136








          • 1




            That might be a reasonable solution. But the filter functions in scipy.signal assume the data is sampled at regular intervals. If you discard the nan values, the sample periods of the remaining data are no longer consistent, and that may invalidate the filter results.
            – Warren Weckesser
            Nov 12 at 14:00










          • In this case one would have to first manually resample the data at a lower frequency to match the removal of NaN values ?
            – Patol75
            Nov 12 at 14:39














          • 1




            That might be a reasonable solution. But the filter functions in scipy.signal assume the data is sampled at regular intervals. If you discard the nan values, the sample periods of the remaining data are no longer consistent, and that may invalidate the filter results.
            – Warren Weckesser
            Nov 12 at 14:00










          • In this case one would have to first manually resample the data at a lower frequency to match the removal of NaN values ?
            – Patol75
            Nov 12 at 14:39








          1




          1




          That might be a reasonable solution. But the filter functions in scipy.signal assume the data is sampled at regular intervals. If you discard the nan values, the sample periods of the remaining data are no longer consistent, and that may invalidate the filter results.
          – Warren Weckesser
          Nov 12 at 14:00




          That might be a reasonable solution. But the filter functions in scipy.signal assume the data is sampled at regular intervals. If you discard the nan values, the sample periods of the remaining data are no longer consistent, and that may invalidate the filter results.
          – Warren Weckesser
          Nov 12 at 14:00












          In this case one would have to first manually resample the data at a lower frequency to match the removal of NaN values ?
          – Patol75
          Nov 12 at 14:39




          In this case one would have to first manually resample the data at a lower frequency to match the removal of NaN values ?
          – Patol75
          Nov 12 at 14:39


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53262014%2fhow-to-apply-scipy-signal-filtfilt-on-incomplete-data%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Guess what letter conforming each word

          Port of Spain

          Run scheduled task as local user group (not BUILTIN)