You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 05:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Jan 01, 2026 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
197399930,9rtkit-daemon1972-21rtkit-daemon20:41:152
197399930,80rtkit-daemon0-21swapper/219:09:282
270358028140,14sleep00-21swapper/019:06:190
27737692710,5sleep3411rcuc/322:25:133
270357326841,13sleep30-21swapper/319:06:143
27682642670,3sleep20-21swapper/222:10:032
270238126041,13sleep10-21swapper/119:05:031
270400299581,12cyclictest2143-21upowerd21:34:242
2704003995626,26cyclictest2712795-21python319:30:283
197399530,26rtkit-daemon4497-21PK-Backend00:15:303
2703994995215,7cyclictest141rcu_preempt19:55:270
197399520,8rtkit-daemon1972-21rtkit-daemon21:14:310
197399520,25rtkit-daemon0-21swapper/323:06:173
27047342500,14sleep1261ksoftirqd/119:10:201
27428632490,8sleep10-21swapper/120:55:281
27950092480,6sleep3411rcuc/323:25:123
2704003994835,8cyclictest2736529-21grep20:39:593
2704003994835,8cyclictest2736529-21grep20:39:593
270400399461,32cyclictest2727763-21taskset20:13:443
270400399461,32cyclictest2727763-21taskset20:13:443
2703994994514,26cyclictest2789498-21turbostat23:10:000
197399450,31rtkit-daemon4497-21PK-Backend23:09:592
270400399446,21cyclictest42-21ksoftirqd/320:30:013
197399440,8rtkit-daemon34-21ksoftirqd/221:39:252
27528552430,6sleep00-21swapper/021:25:170
2703998994326,13cyclictest1-21systemd23:00:001
2703998994326,13cyclictest1-21systemd23:00:001
27413452420,11sleep20-21swapper/220:50:302
27348792420,6sleep34497-21PK-Backend20:35:023
270400399421,19cyclictest42-21ksoftirqd/321:34:233
2703994994223,16cyclictest2781274-21cpuspeed_turbos22:45:140
197399420,29rtkit-daemon0-21swapper/323:55:123
197399420,29rtkit-daemon0-21swapper/323:55:123
27082732410,15sleep126-21ksoftirqd/119:20:151
270400399415,17cyclictest42-21ksoftirqd/300:05:013
2704003994123,14cyclictest2793588-21df23:20:153
270400399411,36cyclictest2724098-21cron20:05:013
2703998994126,11cyclictest2778814-21apt-get22:40:001
2703998994122,6cyclictest26-21ksoftirqd/119:55:191
197399410,6rtkit-daemon0-21swapper/319:25:033
2704003994019,17cyclictest2761141-21unin-run21:50:013
2704003994019,17cyclictest2761141-21unin-run21:50:013
270400399401,14cyclictest4497-21PK-Backend00:29:383
2704003994010,16cyclictest42-21ksoftirqd/323:14:153
2704002994025,11cyclictest2814673-21cron00:25:002
270399899402,8cyclictest2715430-21munin-run19:40:001
2703998994021,6cyclictest26-21ksoftirqd/121:00:441
2703998994021,6cyclictest26-21ksoftirqd/121:00:441
2703998994018,18cyclictest2819938-21latency_hist00:40:021
2703994994026,10cyclictest13-21ksoftirqd/021:30:250
197399400,27rtkit-daemon4497-21PK-Backend23:39:163
197399400,27rtkit-daemon4497-21PK-Backend21:43:182
197399400,26rtkit-daemon0-21swapper/221:29:512
2704003993928,8cyclictest42-21ksoftirqd/322:12:593
2704003993928,8cyclictest42-21ksoftirqd/322:02:133
2704003993915,10cyclictest42-21ksoftirqd/320:54:103
2704002993924,11cyclictest923-21dbus-daemon23:19:562
270400299391,22cyclictest2768472-21grep22:10:152
2703998993923,12cyclictest923-21dbus-daemon21:15:011
2703998993922,5cyclictest26-21ksoftirqd/119:33:191
2703998993921,6cyclictest26-21ksoftirqd/123:31:431
2703998993921,6cyclictest26-21ksoftirqd/121:49:281
2703998993921,6cyclictest26-21ksoftirqd/121:49:281
2703994993920,14cyclictest2811815-21df_inode00:15:160
197399390,5rtkit-daemon0-21swapper/222:42:012
197399390,27rtkit-daemon4497-21PK-Backend23:17:103
197399390,27rtkit-daemon4497-21PK-Backend20:44:483
2704003993828,6cyclictest2784552-21/usr/sbin/munin22:55:173
2704003993828,6cyclictest2784552-21/usr/sbin/munin22:55:163
2704003993822,11cyclictest438-20systemd-journal21:26:283
270400399381,33cyclictest42-21ksoftirqd/321:35:123
270400399381,15cyclictest2773307-21ls22:25:003
2704003993811,15cyclictest42-21ksoftirqd/321:09:113
2704002993822,12cyclictest2048-21gnome-session-b22:02:112
2704002993822,12cyclictest1-21systemd19:55:002
2703998993824,10cyclictest2724168-21mktemp20:05:011
2703998993823,11cyclictest923-21dbus-daemon21:34:231
2703998993821,5cyclictest26-21ksoftirqd/122:15:111
2703994993827,8cyclictest2780857-21idleruntime-cro22:45:000
2703994993820,13cyclictest2795711-21ntpq23:25:250
2703994993820,12cyclictest2769670-21libinput-device22:13:010
2703994993819,16cyclictest2705699-21unin-run19:15:010
197399380,27rtkit-daemon4497-21PK-Backend00:11:383
197399380,26rtkit-daemon0-21swapper/221:57:452
197399380,25rtkit-daemon0-21swapper/223:34:062
197399380,25rtkit-daemon0-21swapper/221:50:132
27171732370,6sleep02717157-21munin-run19:45:010
2704003993724,10cyclictest42-21ksoftirqd/319:55:423
2704003993721,12cyclictest2780473-40apt-check22:40:563
2704003993721,12cyclictest2731323-21cron20:25:003
270400299372,7cyclictest2725390-21ntp_states20:05:252
270400299372,12cyclictest2781868-21ntp_states22:45:252
2704002993718,15cyclictest2708530-21df_inode19:20:172
270400299371,14cyclictest2819889-21idleruntime-cro00:40:002
2704002993711,11cyclictest34-21ksoftirqd/222:55:432
2704002993711,11cyclictest34-21ksoftirqd/222:55:422
2703998993727,6cyclictest2736520-21cron20:40:001
2703998993727,6cyclictest2736520-21cron20:39:591
2703998993720,13cyclictest2807724-21sed00:00:241
2703998993719,14cyclictest2817055-21df_inode00:30:151
2703998993719,14cyclictest2817055-21df_inode00:30:151
270399499372,8cyclictest2778644-21list-oem-metapa22:38:230
2703994993720,12cyclictest438-20systemd-journal22:24:280
2703994993720,12cyclictest438-20systemd-journal21:19:560
2703994993719,14cyclictest2713424-21munin-run19:35:020
2703994993719,13cyclictest2757346-21apt-get21:40:000
2703994993718,14cyclictest2760398-21ntp_states21:45:260
2703994993718,14cyclictest2760398-21ntp_states21:45:250
2703994993715,18cyclictest2734882-21latency_hist20:35:010
270399499371,33cyclictest0-21swapper/022:34:200
197399370,27rtkit-daemon4497-21PK-Backend19:36:362
197399370,26rtkit-daemon4497-21PK-Backend22:51:513
197399370,26rtkit-daemon4497-21PK-Backend19:12:302
197399370,26rtkit-daemon0-21swapper/322:18:103
197399370,25rtkit-daemon4497-21PK-Backend23:46:493
197399370,25rtkit-daemon4497-21PK-Backend23:46:483
197399370,25rtkit-daemon4497-21PK-Backend20:07:263
197399370,24rtkit-daemon0-21swapper/321:57:593
270400399365,19cyclictest141rcu_preempt21:50:203
270400399362,12cyclictest438-20systemd-journal23:54:593
2704003993620,12cyclictest2704517-21expr19:10:153
270400299368,7cyclictest141rcu_preempt19:35:012
270400299366,14cyclictest2773784-21cstates22:25:132
2704002993625,8cyclictest34-21ksoftirqd/219:55:412
2704002993622,10cyclictest2799868-21cron23:40:002
2704002993622,10cyclictest1-21systemd00:35:012
2704002993622,10cyclictest1-21systemd00:35:012
270400299362,19cyclictest0-21swapper/222:15:182
2704002993621,11cyclictest2779062-21rm22:40:022
2704002993620,12cyclictest2805612-21cat23:55:122
2704002993620,12cyclictest2805612-21cat23:55:122
2704002993620,12cyclictest2749292-21diskstats21:15:182
2704002993620,12cyclictest2345-21gdbus00:09:572
270400299361,12cyclictest2717392-21/usr/sbin/munin19:45:262
270400299361,12cyclictest2717392-21/usr/sbin/munin19:45:252
2703998993624,8cyclictest2734790-21cron20:35:001
2703998993624,8cyclictest2069-21gnome-shell21:50:101
2703998993624,7cyclictest2736984-21tr20:40:141
2703998993622,11cyclictest2811015-21systemd00:15:011
2703998993622,10cyclictest26-21ksoftirqd/121:41:021
2703998993620,12cyclictest2812447-21sed00:15:311
2703998993619,12cyclictest438-20systemd-journal22:35:011
2703998993618,14cyclictest2805708-21df_inode23:55:141
2703998993618,14cyclictest2805708-21df_inode23:55:141
2703994993620,12cyclictest2774755-21sshd22:26:350
2703994993618,13cyclictest438-20systemd-journal22:00:000
2703994993618,13cyclictest2782849-21apt22:50:130
2703994993618,13cyclictest2739284-21ntpq20:45:260
270399499361,5cyclictest4497-21PK-Backend21:50:400
270399499361,32cyclictest0-21swapper/020:35:430
270399499361,32cyclictest0-21swapper/020:35:430
270399499361,31cyclictest4497-21PK-Backend23:33:190
270399499361,31cyclictest4497-21PK-Backend22:02:060
197399360,24rtkit-daemon0-21swapper/223:23:292
270400399354,8cyclictest42-21ksoftirqd/321:40:413
2704003993522,9cyclictest2715241-40lsb_release19:38:053
2704003993519,12cyclictest2787040-21ntpq23:00:243
2704003993512,19cyclictest2768103-21unin-run22:10:023
270400399351,17cyclictest4497-21PK-Backend22:35:253
270400299352,17cyclictest0-21swapper/219:15:282
270400299352,12cyclictest2772717-21python322:20:282
2704002993519,12cyclictest2787065-21ntpq23:00:272
2704002993518,13cyclictest2746037-21memory21:05:212
270400299351,30cyclictest2715951-21expr19:40:202
270400299351,10cyclictest4497-21PK-Backend21:45:212
270400299351,10cyclictest4497-21PK-Backend21:45:202
2703998993521,10cyclictest2726134-21cron20:10:001
2703998993520,11cyclictest2809219-21systemd00:10:011
2703998993520,11cyclictest2756505-21grep21:35:231
2703998993519,12cyclictest2741361-21seq20:50:321
2703998993519,12cyclictest1036-21gdbus00:29:561
2703998993519,11cyclictest2813844-21ls00:20:231
2703998993518,12cyclictest2781105-21find22:45:131
2703998993518,12cyclictest2276-21gsd-sharing22:02:101
2703998993516,14cyclictest1975-21dbus-daemon22:13:011
270399899351,4cyclictest4497-21PK-Backend23:29:281
2703998993511,7cyclictest27509462expr21:20:161
2703994993523,8cyclictest2746204-21ntp_states21:05:250
2703994993523,7cyclictest2724773-21tr20:05:130
2703994993521,10cyclictest2803330-21cron23:49:590
2703994993521,10cyclictest2803330-21cron23:49:590
2703994993520,11cyclictest2766753-21cut22:05:140
2703994993516,6cyclictest2738249-21cron20:45:000
270399499351,4cyclictest4497-21PK-Backend19:38:130
270399499351,31cyclictest4497-21PK-Backend19:17:500
270399499351,31cyclictest0-21swapper/023:44:490
270399499351,30cyclictest0-21swapper/000:35:230
197399350,24rtkit-daemon4497-21PK-Backend00:19:302
197399350,24rtkit-daemon0-21swapper/200:10:502
270400399342,15cyclictest0-21swapper/323:30:263
2704003993420,10cyclictest42-21ksoftirqd/300:07:413
2704003993419,10cyclictest42-21ksoftirqd/321:00:253
2704003993419,10cyclictest42-21ksoftirqd/321:00:253
2704003993415,6cyclictest141rcu_preempt20:15:143
2704003993414,6cyclictest141rcu_preempt21:00:013
270400399341,28cyclictest2717043-21taskset19:43:483
270400399341,14cyclictest42-21ksoftirqd/319:55:023
2704002993423,6cyclictest34-21ksoftirqd/223:25:122
2704002993419,11cyclictest2802358-21awk23:45:202
2704002993419,11cyclictest2802358-21awk23:45:192
2704002993418,12cyclictest976-21rs:main2
2704002993418,12cyclictest2816189-21seq00:27:192
270400299341,14cyclictest2735252-21tr20:35:162
270400299341,14cyclictest2735252-21tr20:35:162
2703998993422,8cyclictest2069-21gnome-shell22:25:091
2703998993421,9cyclictest2750388-21munin-plugin-st21:20:001
2703998993419,11cyclictest2802149-21expr23:45:131
2703998993419,11cyclictest2802149-21expr23:45:131
2703998993419,11cyclictest2799348-21cat23:35:281
2703998993418,12cyclictest923-21dbus-daemon21:30:001
2703998993417,7cyclictest2803652-21cpu23:50:151
2703998993417,13cyclictest2746317-21sensors_temp21:05:281
2703998993417,12cyclictest466-21systemd-udevd19:25:411
270399899341,12cyclictest2719043-21cat19:50:021
270399899341,12cyclictest2719043-21cat19:50:021
2703994993420,10cyclictest2742288-21expr20:55:140
2703994993420,10cyclictest2741344-21/usr/sbin/munin20:50:310
2703994993419,11cyclictest2798867-21cat23:35:190
2703994993419,11cyclictest2345-21gdbus00:29:570
2703994993418,11cyclictest2810907-21apt-get00:15:010
270399499341,5cyclictest4497-21PK-Backend00:03:580
270399499341,4cyclictest0-21swapper/000:32:230
270399499341,4cyclictest0-21swapper/000:32:230
270399499341,30cyclictest4497-21PK-Backend21:01:290
270399499341,30cyclictest4497-21PK-Backend21:01:290
2704003993321,7cyclictest2706787-21df19:15:143
270400399331,5cyclictest4497-21PK-Backend22:45:253
2704003993313,5cyclictest141rcu_preempt22:32:333
270400399331,14cyclictest4497-21PK-Backend00:20:213
2704002993323,5cyclictest34-21ksoftirqd/220:00:532
2704002993322,6cyclictest34-21ksoftirqd/222:30:222
2704002993319,10cyclictest2748763-21grep21:15:012
2704002993318,11cyclictest2731516-21sed20:25:012
2704002993317,12cyclictest2751927-21seq21:22:522
270400299331,29cyclictest0-21swapper/223:41:292
2704002993312,16cyclictest2744574-21cut21:00:282
2704002993312,16cyclictest2744574-21cut21:00:282
270400299331,10cyclictest34-21ksoftirqd/223:10:512
2703998993319,10cyclictest2719458-21expr19:50:131
2703998993318,11cyclictest2784257-21seq22:54:311
2703998993318,11cyclictest2771932-21cpuspeed22:20:141
2703998993318,11cyclictest2766665-21sort22:05:121
2703998993317,12cyclictest2801489-21sshd23:43:371
2703998993315,13cyclictest2788424-21diskstats23:05:231
270399899331,29cyclictest4497-21PK-Backend21:55:401
270399899331,29cyclictest4497-21PK-Backend20:21:311
2703998993310,6cyclictest141rcu_preempt19:40:211
2703994993323,6cyclictest2793351-21cpuspeed_turbos23:20:140
2703994993318,11cyclictest2814254-21sed00:20:310
2703994993317,11cyclictest2792222-21ntpq23:15:260
2703994993317,11cyclictest2758459-21cut21:40:220
270399499331,30cyclictest0-21swapper/023:14:330
270399499331,29cyclictest0-21swapper/021:21:390
270399499331,29cyclictest0-21swapper/019:22:110
270399499331,28cyclictest2728399-21expr20:15:150
270399499331,28cyclictest0-21swapper/020:23:160
270399499331,10cyclictest4497-21PK-Backend22:55:430
270399499331,10cyclictest4497-21PK-Backend22:55:420
270400399326,7cyclictest42-21ksoftirqd/320:45:173
270400399326,4cyclictest141rcu_preempt21:15:333
2704003993220,9cyclictest42-21ksoftirqd/319:20:143
2704003993218,10cyclictest2751267-21wc21:20:253
2704003993218,10cyclictest2747479-21expr21:10:143
2704003993217,11cyclictest2819377-21grep00:35:303
270400399321,16cyclictest4497-21PK-Backend23:40:303
270400399321,11cyclictest2717812-21fw_packets19:45:173
270400399321,11cyclictest2717812-21fw_packets19:45:173
2704002993220,8cyclictest2069-21gnome-shell20:56:102
2704002993218,10cyclictest2784339-21systemctl22:55:002
2704002993217,11cyclictest2738597-21cat20:45:132
2704002993217,11cyclictest2732319-21grep20:25:242
2704002993217,11cyclictest2710362-21cstates19:25:162
2704002993216,11cyclictest438-20systemd-journal00:05:002
2703998993225,4cyclictest4497-21PK-Backend23:01:391
2703998993220,8cyclictest2707899-21ls19:20:001
2703998993218,10cyclictest2791749-21ls23:15:161
2703998993217,11cyclictest2726660-21expr20:10:171
2703998993217,11cyclictest2726660-21expr20:10:171
2703998993217,10cyclictest2789785-21dpkg23:10:271
2703998993216,12cyclictest2780762-21seq22:44:051
2703998993215,12cyclictest2738755-21expr20:45:141
2703998993215,12cyclictest2728517-21df_inode20:15:171
270399499324,23cyclictest2803756-21tr23:50:150
2703994993217,11cyclictest2727599-21seq20:11:360
2703994993217,11cyclictest2727599-21seq20:11:360
2703994993215,12cyclictest438-20systemd-journal20:03:280
270399499321,4cyclictest4497-21PK-Backend23:58:380
270399499321,4cyclictest4497-21PK-Backend23:58:370
270399499321,4cyclictest4497-21PK-Backend19:54:300
2703994993214,13cyclictest2770825-21ntpq22:15:230
270400399318,6cyclictest141rcu_preempt00:30:303
270400399318,6cyclictest141rcu_preempt00:30:293
2704002993117,10cyclictest2803616-21seq23:50:142
2704002993115,11cyclictest2728964-21processes20:15:262
2703994993117,10cyclictest2731737-21grep20:25:150
270399499311,28cyclictest0-21swapper/019:29:030
270399499311,25cyclictest2787983-21sed23:05:010
2704002993017,9cyclictest2726550-21cut20:10:172
2704002993017,9cyclictest2726550-21cut20:10:162
2703998993015,10cyclictest2794917-21date23:25:021
270399499301,4cyclictest0-21swapper/019:46:530
270399499301,4cyclictest0-21swapper/019:46:530
270399899281,12cyclictest141rcu_preempt20:29:321
2703994992815,9cyclictest959-21udisksd00:09:560
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional