Announcement

Collapse
No announcement yet.

Using zgrep to look at dpkg logs

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    [SOLVED] Using zgrep to look at dpkg logs

    kubicle mentioned the use of zgrep here in the context of searching dpkg's logs.

    I use this small yad script

    Code:
    #!/bin/bash
    package_name="$(yad --center --width=300 --title "Search dpkg logs" --entry --entry-label="Enter the exact package_name:" 2>/dev/null)"
    zgrep -E "status (not-)?installed $package_name:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t
    which, thanks to input from jlittle and kubicle, is much better than what I posted originally!


    Code:
    #!/bin/bash
    package_name="$(yad --center --width=300 --title "Search dpkg logs" --entry --entry-label="Enter the exact package_name:" 2>/dev/null)"
    zgrep -e "status installed $package_name:" /var/log/dpkg.log* > /tmp/zgrep.txt
    zgrep -e "status not-installed $package_name:" /var/log/dpkg.log* >> /tmp/zgrep.txt
    cat /tmp/zgrep.txt | sort -k4
    sleep 1s
    rm /tmp/zgrep.txt


    But
    it requires the exact package name
    and
    the archived logs should still be in /var/log

    To make the second condition more likely, I've bumped up the retention of logs (/var/log/apt/history.log and /var/log/dpkg.log) from the default rotate 12 to rotate 60.
    Last edited by chimak111; May 14, 2020, 07:35 AM.
    Kubuntu 20.04

    #2
    Thank you, I've always just used zcat whatever | grep, but that loses the file name, so I'll try to think of zgrep next time.

    BTW, your script could use a sub-shell:
    Code:
    (  zgrep -e "status installed $package_name:"       /var/log/dpkg.log*;
    zgrep -e "status not-installed $package_name:" /var/log/dpkg.log* ) |
    sort -k4
    but the two searches are so similar,
    Code:
    zgrep -Ee "status (not-)?installed $package_name:" /var/log/dpkg.log* |
    sort -k4
    (An egrep regex, -E, to avoid backslash-itis, they'd have to be doubled)
    Last edited by jlittle; May 13, 2020, 12:43 AM. Reason: avoid bowdler
    Regards, John Little

    Comment


      #3
      Originally posted by jlittle View Post
      Code:
      zgrep -Ee "status (not-)?installed $package_name:" /var/log/dpkg.log* |
      sort -k4
      Thanks for that! Although just "-E" works for me. "-Ee" also works whereas just "-e" does nothing in the current context.

      Now, I have
      Code:
      zgrep -E "status (not-)?installed $package_name:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r
      The sed bit allows sorting by date and time.
      Last edited by chimak111; May 13, 2020, 06:59 AM.
      Kubuntu 20.04

      Comment


        #4
        I only left in the -e because you'd used it. I don't normally, with only one regex, I've seen scripts where the writers like -e with grep, maybe they're big sed users, where multiple -e phrases are common.

        I'd wondered about the -k4. The fourth space-delimited field is always "installed" for me; maybe your date format has more spaces, mine is ISOish with one space.
        Regards, John Little

        Comment


          #5
          This is my "default" output where date is separated by a space from time but there's no space between the file/archive name and date:
          Code:
          $ zgrep -E "status (not-)?installed snapd:" /var/log/dpkg.log*
          /var/log/dpkg.log:2020-05-13 21:53:19 status installed snapd:amd64 2.42.1+18.04
          /var/log/dpkg.log:2020-05-13 21:53:36 status not-installed snapd:amd64 <none>
          /var/log/dpkg.log.24.gz:2018-04-26 18:25:46 status installed snapd:amd64 2.32.5+18.04
          /var/log/dpkg.log.24.gz:2018-05-15 22:48:04 status installed snapd:amd64 2.32.5+18.04
          /var/log/dpkg.log.24.gz:2018-05-15 22:48:10 status not-installed snapd:amd64 <none>
          /var/log/dpkg.log.5.gz:2019-12-11 16:33:51 status installed snapd:amd64 2.42.1+18.04
          $
          And this is the new output with a "sed" step (to introduce a space between the file/archive name and date) followed by "sort -k2,3 -r". Now the listing is sorted better, IMO.

          Code:
          $ yad-zgrep-dpkg-logs.sh
          /var/log/dpkg.log: 2020-05-13 21:53:36 status not-installed snapd:amd64 <none>
          /var/log/dpkg.log: 2020-05-13 21:53:19 status installed snapd:amd64 2.42.1+18.04
          /var/log/dpkg.log.5.gz: 2019-12-11 16:33:51 status installed snapd:amd64 2.42.1+18.04
          /var/log/dpkg.log.24.gz: 2018-05-15 22:48:10 status not-installed snapd:amd64 <none>
          /var/log/dpkg.log.24.gz: 2018-05-15 22:48:04 status installed snapd:amd64 2.32.5+18.04
          /var/log/dpkg.log.24.gz: 2018-04-26 18:25:46 status installed snapd:amd64 2.32.5+18.04
          $
          Kubuntu 20.04

          Comment


            #6
            You could pipe to column for a nicer output, as in:
            Code:
            zgrep -E "status (not-)?installed $package_name:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t
            If you wish to do more advanced output formatting (and/or avoid long pipe chains), then "awk" is your tool (it will be able to all your sed/sort/column in one go, admittedly on a learning curve, though)
            Last edited by kubicle; May 14, 2020, 02:03 AM.

            Comment


              #7
              Originally posted by kubicle View Post
              You could pipe to column for a nicer output, as in:
              Code:
              zgrep -E "status (not-)?installed $package_name:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t
              If you wish to do more advanced output formatting (and/or avoid long pipe chains), then "awk" is your tool (it will be able to all your sed/sort/column in one go, admittedly on a learning curve, though)
              Perfect! (Although your comment about using awk will gnaw away at me)
              Kubuntu 20.04

              Comment


                #8
                Your command string has $package_name (to be replaced by what ever package name you want to search against), but if I keep the $ (say, $snapd) no results are found. If I don't keep the $ (snapd), then the command string works.
                Using Kubuntu Linux since March 23, 2007
                "It is a capital mistake to theorize before one has data." - Sherlock Holmes

                Comment


                  #9
                  Originally posted by Snowhog View Post
                  Your command string has $package_name (to be replaced by what ever package name you want to search against), but if I keep the $ (say, $snapd) no results are found. If I don't keep the $ (snapd), then the command string works.
                  The command is meant to be used in a script (see the first post) where "$package_name" is a variable.

                  Comment


                    #10
                    I don't really want to use yad, so I made a one-liner that accepts the package name as argument (and doesn't need an exact match, partial matches should be fine)

                    Code:
                    #!/bin/bash
                    zgrep -E "status (not-)?installed [^ ]*${1}[^ ]*:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t
                    Usage:
                    command <part_of_packagename>

                    Could be improved with some commandline options (for example, when you actually want to search for an exact match)
                    Last edited by kubicle; May 14, 2020, 08:23 AM.

                    Comment


                      #11
                      Originally posted by kubicle View Post
                      I don't really want to use yad,
                      Since a terminal has to be open in any case I guess yad is unnecessary. But are there any other reasons for not wanting to use yad generally?

                      Originally posted by kubicle View Post
                      so I made a one-liner that accepts the package name as argument (and doesn't need an exact match, partial matches should be fine)

                      Code:
                      #!/bin/bash
                      zgrep -E "status (not-)?installed [^ ]*${1}[^ ]*:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t
                      Usage:
                      command <part_of_packagename>

                      Could be improved with some commandline options (for example, when you actually want to search for an exact match)
                      What does
                      Code:
                      [^ ]*
                      do? Does it stand for a string of any number of non-space characters non-greedily?

                      In the specific context of "status (not-)?installed <string>:",

                      Code:
                      .*${1}.*
                      seems to work just as well.
                      Last edited by chimak111; May 14, 2020, 10:25 AM.
                      Kubuntu 20.04

                      Comment


                        #12
                        Originally posted by chimak111 View Post
                        Since a terminal has to be open in any case I guess yad is unnecessary. But are there any other reasons for not wanting to use yad generally?
                        The main reason is that since it's a cli tool, the switch to gui input is not really to my liking (and if I was to use a gui dialog, I'd prefer "kdialog")

                        Originally posted by chimak111 View Post
                        What does
                        Code:
                        [^ ]*
                        do? Does it stand for a string of any number of non-space characters non-greedily?
                        Yes, it'll match any number of non-space chars

                        Originally posted by chimak111 View Post
                        In the specific context of "status (not-)?installed <string>:",
                        Code:
                        .*${1}.*
                        seems to work just as well.
                        In that specific string it shouldn't make much of difference, it's just a habit of mine to avoid .* wildcards when possible in long line REGEXes, ít's somewhat prone to extensively wide matches. But since the search string is so exact in this script, it shouldn't make a difference in this case. Still, it's sort of future-proofing in case I wish to modify the script more.

                        Comment


                          #13
                          Re. kdialog, I too would prefer it, but it doesn't seem possible to format the windows as well as is possible with yad.

                          And yad has this cute color picker (accessed with yad --color) although I came across mention of a cli color picker which is not available for Ubuntu. I'll open a new thread on that.
                          Kubuntu 20.04

                          Comment


                            #14
                            I added a "test" to ensure a search string is provided:

                            Code:
                            #!/bin/bash
                            echo "enter the_string"
                            read the_string
                            [ "$the_string" ] || { echo "You forgot the search string!" ; exit 1 ; }
                            zgrep -E "status (not-)?installed [^ ]*$the_string[^ ]*:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t
                            Kubuntu 20.04

                            Comment


                              #15
                              Originally posted by kubicle View Post
                              I don't really want to use yad, so I made a one-liner that accepts the package name as argument (and doesn't need an exact match, partial matches should be fine)

                              Code:
                              #!/bin/bash
                              zgrep -E "status (not-)?installed [^ ]*${1}[^ ]*:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t
                              Usage:
                              command <part_of_packagename>

                              Could be improved with some commandline options (for example, when you actually want to search for an exact match)
                              I modified the code further to
                              • exit the script if the user presses Enter without providing a search string
                              • allow for the use of an exact match or part of the package name. For example, entering snapd will provide only snapd results whereas entering .*snapd.* will provide results for libsnapd-glib1 as well


                              Code:
                              #!/bin/bash
                              
                              echo "enter the package name;"; echo "use .* as prefix/suffix if the exact package name is not known"
                              read the_string
                              [ "$the_string" ] || { echo "You forgot the search string!" ; exit 1 ; }
                              zgrep -E "status (not-)?installed $the_string:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t



                              However, a notable failure of using zgrep on dkpg logs appears to relate to the very first (oldest) dpkg.log, the one recording the initial install.

                              For example, the code doesn't show status not-installed for packages such as casper, ubiquity-slideshow-kde, and many more that were automatically purged from the system during the installation process.

                              I've noted this on Kubuntu 18.04. I'll check in some *buntu 20.04 VMs as well.




                              Okay, it's the same in a VM of Kubuntu 20.04.It's as if dpkg is unaware of what is recorded in the corresponding /var/log/apt/history.log in the section beginning with
                              Code:
                              Start-Date: 2020-01-23  16:01:05
                              Requested-By: kubuntu (999)
                              Purge: ...
                              Last edited by chimak111; May 18, 2020, 07:22 AM.
                              Kubuntu 20.04

                              Comment

                              Working...
                              X