You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-13 - 22:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sun Jul 13, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
162399970,5rtkit-daemon0-21swapper/010:41:011
25856022960,6sleep30-21swapper/309:10:293
162399960,5rtkit-daemon2197014-21PK-Backend11:19:421
162399940,6rtkit-daemon2197014-21PK-Backend10:08:531
162399940,6rtkit-daemon2197014-21PK-Backend10:08:531
162399840,10rtkit-daemon1753-21gnome-shell09:41:041
254171928154,20sleep20-21swapper/207:07:152
254167527740,9sleep30-21swapper/307:06:383
254165127539,10sleep10-21swapper/107:06:201
162399740,4rtkit-daemon0-21swapper/307:35:363
26390652730,4sleep20-21swapper/211:40:332
254191127240,26sleep00-21swapper/007:09:410
254206999682,38cyclictest1753-21gnome-shell10:50:122
254206999672,38cyclictest2601516-21/usr/sbin/munin09:55:232
254206999662,46cyclictest1753-21gnome-shell09:21:082
254206999662,37cyclictest1753-21gnome-shell10:05:342
254206999662,37cyclictest1753-21gnome-shell10:05:342
254206999661,37cyclictest2587580-21taskset09:15:432
254206999652,36cyclictest2594791-21/usr/sbin/munin09:35:342
254206999651,40cyclictest860997-21snmpd11:11:312
254206999642,36cyclictest1753-21gnome-shell12:13:422
254206999642,36cyclictest1753-21gnome-shell12:13:412
254206999642,36cyclictest1753-21gnome-shell07:57:462
254206999642,35cyclictest1753-21gnome-shell09:01:572
254206999641,40cyclictest2635658-21taskset11:30:552
254206999641,38cyclictest2557436-21sh07:50:292
254206999641,38cyclictest2543469-21taskset07:11:432
254206999632,37cyclictest1753-21gnome-shell08:24:192
254206999632,37cyclictest1753-21gnome-shell08:24:182
254206999632,35cyclictest1753-21gnome-shell10:14:232
254206999632,34cyclictest2547744-21cat07:25:012
254206999631,39cyclictest1753-21gnome-shell12:05:482
254206999631,36cyclictest2555389-21/usr/sbin/munin07:45:232
254206999631,35cyclictest2357941-21cups-browsed12:00:042
254206999631,34cyclictest860997-21snmpd12:26:342
2542069996313,29cyclictest3242255-21gdbus12:20:022
254206999622,43cyclictest1753-21gnome-shell11:59:062
254206999622,43cyclictest1753-21gnome-shell10:47:152
254206999622,43cyclictest1753-21gnome-shell08:50:552
254206999622,42cyclictest1753-21gnome-shell11:15:112
254206999622,35cyclictest1753-21gnome-shell12:18:442
254206999622,34cyclictest1753-21gnome-shell07:19:392
2542069996212,35cyclictest2598276-21grep09:45:282
2542069996212,35cyclictest2598276-21grep09:45:282
162399620,8rtkit-daemon1622-21rtkit-daemon10:58:412
254206999612,43cyclictest1753-21gnome-shell09:52:352
254206999612,42cyclictest1753-21gnome-shell10:00:092
254206999612,42cyclictest1753-21gnome-shell08:48:562
254206999612,41cyclictest1753-21gnome-shell12:30:072
254206999612,36cyclictest1753-21gnome-shell08:17:302
254206999612,34cyclictest1753-21gnome-shell11:53:372
254206999612,34cyclictest1753-21gnome-shell11:39:092
254206999612,34cyclictest1753-21gnome-shell11:04:062
254206999612,34cyclictest1753-21gnome-shell07:25:452
254206999612,33cyclictest1753-21gnome-shell08:05:482
254206999611,55cyclictest2616889-21cat10:40:132
254206999610,57cyclictest1622-21rtkit-daemon11:46:582
254206999602,42cyclictest1753-21gnome-shell09:08:272
254206999602,35cyclictest1753-21gnome-shell11:28:512
254206999602,33cyclictest1753-21gnome-shell08:41:162
254206999601,36cyclictest2608219-21/usr/sbin/munin10:15:172
254206999601,35cyclictest860997-21snmpd09:27:042
254206999601,34cyclictest2596414-21/usr/sbin/munin09:40:262
254206999601,32cyclictest2612246-21/usr/sbin/munin10:25:302
254206999601,30cyclictest0-21swapper/208:00:122
254206999592,41cyclictest1753-21gnome-shell11:24:212
254206999592,41cyclictest1753-21gnome-shell10:37:502
254206999592,41cyclictest1753-21gnome-shell08:27:282
254206999592,34cyclictest1753-21gnome-shell12:37:282
254206999592,34cyclictest1753-21gnome-shell11:05:562
254206999592,34cyclictest1753-21gnome-shell11:05:562
254206999592,34cyclictest1753-21gnome-shell10:34:082
254206999592,34cyclictest1753-21gnome-shell07:31:292
254206999592,33cyclictest1753-21gnome-shell10:24:402
254206999592,33cyclictest1753-21gnome-shell08:31:042
254206999592,32cyclictest1753-21gnome-shell08:37:212
254206999582,34cyclictest1753-21gnome-shell07:39:102
2542065995825,29cyclictest2649062-21grep12:10:261
2542065995825,29cyclictest2649062-21grep12:10:261
162399580,9rtkit-daemon1622-21rtkit-daemon08:59:293
254206999572,34cyclictest1753-21gnome-shell09:33:312
162399570,8rtkit-daemon1622-21rtkit-daemon09:00:253
254206999562,32cyclictest1753-21gnome-shell07:44:482
26577182530,6chrt2197014-21PK-Backend12:35:160
2542065995316,13cyclictest2607568-21cron10:15:001
254206599531,25cyclictest2650230-21cpuspeed_turbos12:15:171
2542069995212,35cyclictest2579756-21expr08:55:152
2542065995220,27cyclictest2646534-21date12:05:021
2542065995214,13cyclictest2654189-21cat12:25:161
2542065995123,24cyclictest2635168-21grep11:30:241
2542065995119,28cyclictest855-21dbus-daemon09:04:191
2542065995119,27cyclictest2590865-21sed09:25:351
2542065995115,11cyclictest2618918-21expr10:45:181
26202872500,5chrt251rcuc/110:50:051
2542069995016,29cyclictest2584944-21cstates09:10:162
2542065995017,10cyclictest2567215-21tr08:20:161
2542065995017,10cyclictest2567215-21tr08:20:151
2542065994920,25cyclictest2651925-21apt-key12:20:031
2542065994919,26cyclictest2638896-21sort11:40:321
2542065994919,26cyclictest1069-21runrttasks10:03:291
2542065994917,10cyclictest2657381-21cpu12:35:141
254206599491,14cyclictest2572002-21latency_hist08:35:021
162399490,34rtkit-daemon2197014-21PK-Backend10:40:200
254206599489,16cyclictest1753-21gnome-shell07:20:301
254206599482,8cyclictest2641193-21munin-run11:50:001
254206599482,11cyclictest2569776-21sed08:25:311
254206599482,10cyclictest1753-21gnome-shell12:05:131
254206599481,8cyclictest2607984-21tr10:15:161
254206599481,23cyclictest2594191-21grep09:35:241
2542065994812,14cyclictest438-20systemd-journal09:30:281
2542065994811,11cyclictest2577107-21cat08:45:321
2542073994726,16cyclictest3242253-21nautilus10:40:013
2542073994725,16cyclictest42-21ksoftirqd/307:45:243
254206999471,42cyclictest1069-21runrttasks08:11:402
254206999471,42cyclictest1069-21runrttasks08:11:402
254206599479,11cyclictest2577841-21cat08:50:161
254206599472,10cyclictest2602150-21seq09:57:071
2542065994719,24cyclictest2609781-21cpuspeed_turbos10:20:151
2542065994718,24cyclictest26-21ksoftirqd/107:10:161
2542065994714,10cyclictest2597605-21tr09:45:141
2542065994714,10cyclictest2597605-21tr09:45:141
2542065994713,10cyclictest2563843-21expr08:10:141
2542065994713,10cyclictest2563843-21expr08:10:141
2542065994712,12cyclictest2656084-21cat12:30:201
2542065994712,12cyclictest2575019-21cat08:40:351
254206599469,15cyclictest1753-21gnome-shell11:35:241
2542065994619,23cyclictest2644649-21idleruntime-cro12:00:001
2542065994618,10cyclictest26-21ksoftirqd/107:37:241
2542065994616,10cyclictest26-21ksoftirqd/108:55:031
254206599461,28cyclictest0-21swapper/107:26:411
2542065994612,11cyclictest2544953-21if_enp2s007:15:231
2542065994612,10cyclictest2553476-21expr07:40:161
254206599461,18cyclictest2613498-21chrt10:30:201
254206599461,11cyclictest2624688-21grep11:00:281
2542065994611,11cyclictest2566011-21grep08:15:261
162399460,32rtkit-daemon2197014-21PK-Backend11:20:210
254206599458,11cyclictest0-21swapper/111:05:251
254206599458,11cyclictest0-21swapper/111:05:241
254206599452,7cyclictest2570911-21diskstats08:30:171
254206599452,16cyclictest1753-21gnome-shell11:20:221
2542065994518,23cyclictest2622182-21cpuspeed_turbos10:55:151
254206599451,14cyclictest1753-21gnome-shell11:25:301
2542073994426,13cyclictest438-20systemd-journal12:10:103
2542073994426,13cyclictest438-20systemd-journal12:10:103
2542073994423,17cyclictest2639486-21unin-run11:45:013
254207399442,18cyclictest1787-21gdbus10:16:013
2542065994417,8cyclictest2584012-21timerandwakeup09:05:311
2542065994416,8cyclictest1753-21gnome-shell09:14:351
2542065994414,7cyclictest1753-21gnome-shell10:37:071
254206599441,16cyclictest2586854-21/usr/sbin/munin09:15:211
2542065994410,10cyclictest2550418-21ntp_kernel_pll_07:30:251
162399440,31rtkit-daemon2197014-21PK-Backend11:16:270
254206599438,16cyclictest1753-21gnome-shell07:56:191
2542065994315,6cyclictest231irq_work/110:25:151
2542065994313,8cyclictest0-21swapper/111:10:211
2542065994311,8cyclictest0-21swapper/109:22:321
2542065994310,8cyclictest2600234-21timerwakeupswit09:50:371
2542062994318,21cyclictest2576768-40lsb_release08:45:260
254206299431,7cyclictest2197014-21PK-Backend11:59:510
254207399421,9cyclictest2197014-21PK-Backend12:28:023
2542065994213,9cyclictest64850irq/126-i91508:10:021
2542065994213,6cyclictest231irq_work/107:48:361
2542065994211,6cyclictest0-21swapper/107:51:551
254206299422,7cyclictest2589668-21munin-run09:25:010
162399420,30rtkit-daemon2197014-21PK-Backend10:38:260
162399420,28rtkit-daemon13-21ksoftirqd/009:42:540
254207399416,30cyclictest1753-21gnome-shell08:41:553
2542073994128,9cyclictest1069-21runrttasks08:24:103
2542073994128,9cyclictest1069-21runrttasks08:24:103
2542073994123,13cyclictest2635247-21ntp_kernel_pll_11:30:263
2542073994111,11cyclictest2641906-21if_enp2s011:50:203
254206599419,8cyclictest0-21swapper/108:02:571
254206599411,17cyclictest1753-21gnome-shell11:55:011
2542062994127,6cyclictest2600663-21idleruntime-cro09:55:000
254206299412,34cyclictest1753-21gnome-shell11:34:470
162399410,29rtkit-daemon2197014-21PK-Backend12:29:580
254207399403,32cyclictest1753-21gnome-shell10:05:333
254207399403,32cyclictest1753-21gnome-shell10:05:323
254206299402,7cyclictest1753-21gnome-shell07:11:280
2542062994021,14cyclictest1632-21dbus-daemon11:30:010
162399400,28rtkit-daemon2197014-21PK-Backend07:58:470
2542073993928,7cyclictest2612816-21ls10:30:003
254207399392,32cyclictest1753-21gnome-shell07:25:183
2542073993922,13cyclictest2616813-21apt10:40:113
2542073993918,16cyclictest2651141-21expr12:15:333
2542062993923,12cyclictest2550283-21sshd07:30:230
162399390,27rtkit-daemon2197014-21PK-Backend08:00:410
162399390,26rtkit-daemon2197014-21PK-Backend09:08:420
25689412380,8sleep013-21ksoftirqd/008:25:150
254207399382,8cyclictest1753-21gnome-shell11:49:513
254207399382,7cyclictest2603145-21irqstats10:00:243
2542073993824,10cyclictest42-21ksoftirqd/310:10:163
254207399382,32cyclictest1753-21gnome-shell08:51:363
2542073993821,12cyclictest438-20systemd-journal12:06:093
2542073993821,12cyclictest438-20systemd-journal09:40:023
2542073993821,12cyclictest2599923-21ntpq09:50:313
2542073993818,15cyclictest1-21systemd07:20:013
2542073993815,12cyclictest1738-21gdbus07:41:563
2542073993814,6cyclictest391irq_work/311:13:193
254206299382,13cyclictest1753-21gnome-shell11:42:300
162399380,31rtkit-daemon0-21swapper/309:23:063
162399380,27rtkit-daemon2197014-21PK-Backend08:09:100
2542073993722,11cyclictest2595178-21perf09:40:003
2542073993720,12cyclictest438-20systemd-journal11:00:003
2542073993718,15cyclictest1753-21gnome-shell08:02:203
254207399371,17cyclictest1753-21gnome-shell09:45:483
254207399371,17cyclictest1753-21gnome-shell09:45:483
254206299374,28cyclictest2647736-21users12:05:320
2542062993727,5cyclictest2197014-21PK-Backend08:53:230
2542062993721,11cyclictest2575625-21wc08:45:020
2542062993720,12cyclictest2619376-21awk10:45:310
254206299371,31cyclictest2197014-21PK-Backend08:55:240
162399370,25rtkit-daemon0-21swapper/009:16:260
26518392360,6sleep32197014-21PK-Backend12:20:023
2542073993624,7cyclictest2658804-21sh12:40:003
2542073993622,9cyclictest2592041-21cpuspeed_turbos09:30:163
2542073993621,11cyclictest855-21dbus-daemon11:25:013
2542073993619,13cyclictest855-21dbus-daemon11:05:013
2542073993619,13cyclictest2563340-21/usr/sbin/munin08:10:013
2542073993618,13cyclictest2645636-21ntpq12:00:253
2542062993627,5cyclictest2197014-21PK-Backend11:07:470
2542062993627,5cyclictest2197014-21PK-Backend11:07:470
254206299362,5cyclictest0-21swapper/012:18:120
2542062993625,7cyclictest2645111-21tr12:00:150
254206299362,31cyclictest0-21swapper/008:16:120
2542062993620,12cyclictest2582733-21munin-run09:05:010
254206299361,8cyclictest0-21swapper/009:26:450
254206299361,31cyclictest2197014-21PK-Backend09:36:530
254206299361,30cyclictest2197014-21PK-Backend07:42:540
254206299361,10cyclictest2197014-21PK-Backend09:12:240
2542073993522,9cyclictest1753-21gnome-shell07:21:343
2542073993522,10cyclictest42-21ksoftirqd/311:09:263
2542073993522,10cyclictest42-21ksoftirqd/311:09:263
254207399352,13cyclictest2569170-21df_inode08:25:163
2542073993521,11cyclictest42-21ksoftirqd/311:15:233
2542073993521,11cyclictest42-21ksoftirqd/308:15:273
2542073993520,10cyclictest2636266-21apt11:35:143
2542073993519,12cyclictest2583651-21cat09:05:243
2542073993519,11cyclictest2655689-21cat12:30:133
2542073993516,15cyclictest3242255-21gdbus08:00:013
254207399351,30cyclictest1753-21gnome-shell09:56:563
254207399351,29cyclictest1753-21gnome-shell08:48:023
254206299354,10cyclictest141rcu_preempt11:00:260
254206299352,7cyclictest1753-21gnome-shell08:35:460
2542062993526,5cyclictest2197014-21PK-Backend10:10:480
2542062993520,11cyclictest2642330-21python311:50:290
254206299351,9cyclictest2197014-21PK-Backend11:13:190
254206299351,9cyclictest2197014-21PK-Backend10:03:200
2542062993518,13cyclictest2611094-21/usr/sbin/munin10:25:010
254206299351,4cyclictest2197014-21PK-Backend10:25:540
254206299351,11cyclictest2552935-21cron07:40:000
2542073993425,5cyclictest2197014-21PK-Backend08:35:393
2542073993422,8cyclictest1753-21gnome-shell10:30:083
2542073993422,8cyclictest1753-21gnome-shell09:30:003
2542073993420,9cyclictest1753-21gnome-shell07:30:083
2542073993419,11cyclictest2632555-21cat11:25:163
2542073993419,11cyclictest2610396-21awk10:20:283
2542073993417,12cyclictest438-20systemd-journal08:34:563
254207399341,29cyclictest1753-21gnome-shell07:13:233
254207399341,14cyclictest1734-21gvfs-udisks2-vo08:11:573
254207399341,14cyclictest1734-21gvfs-udisks2-vo08:11:563
254206299342,8cyclictest2637735-21munin-run11:40:000
2542062993425,5cyclictest2197014-21PK-Backend10:54:470
2542062993423,7cyclictest2597442-21/usr/sbin/munin09:45:320
2542062993423,7cyclictest2597442-21/usr/sbin/munin09:45:310
2542062993422,8cyclictest1753-21gnome-shell07:45:350
2542062993421,9cyclictest13-21ksoftirqd/007:15:130
2542062993418,11cyclictest2652758-21cat12:20:240
2542062993415,14cyclictest2571284-21ntpq08:30:260
254206299341,4cyclictest2197014-21PK-Backend09:33:490
254207399332,9cyclictest1753-21gnome-shell09:15:183
2542073993321,8cyclictest2557102-21sort07:50:233
2542073993320,9cyclictest42-21ksoftirqd/310:54:493
2542073993319,10cyclictest2643574-21cat11:55:183
254207399331,13cyclictest1753-21gnome-shell10:45:503
2542062993325,5cyclictest2197014-21PK-Backend08:12:280
2542062993325,5cyclictest2197014-21PK-Backend08:12:280
2542062993324,5cyclictest2197014-21PK-Backend10:15:280
2542062993324,5cyclictest2197014-21PK-Backend10:09:370
2542062993324,5cyclictest2197014-21PK-Backend10:09:360
2542062993322,6cyclictest13-21ksoftirqd/010:31:320
254206299332,12cyclictest1753-21gnome-shell10:56:330
254206299331,9cyclictest2197014-21PK-Backend12:11:430
254206299331,9cyclictest2197014-21PK-Backend12:11:430
254206299331,4cyclictest2197014-21PK-Backend08:23:130
254206299331,4cyclictest2197014-21PK-Backend08:23:130
254206299331,4cyclictest2197014-21PK-Backend07:50:550
254206299331,28cyclictest2197014-21PK-Backend12:33:290
2542062993222,6cyclictest2601805-21python309:55:300
2542062993222,5cyclictest2197014-21PK-Backend11:45:340
254206299312,6cyclictest2549429-21turbostat07:30:000
2542062993121,6cyclictest2547689-21turbostat.cron07:25:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional