You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 10:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Nov 30, 2025 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
246029021100,6sleep20-21swapper/220:25:312
1623991050,9rtkit-daemon881143-21gmain20:40:573
162399890,5rtkit-daemon0-21swapper/323:40:203
162399840,5rtkit-daemon0-21swapper/319:41:383
162399810,5rtkit-daemon0-21swapper/321:58:083
162399810,5rtkit-daemon0-21swapper/321:48:293
162399810,10rtkit-daemon1753-21gnome-shell23:17:093
24842252790,4chrt2483585-21/usr/sbin/munin21:35:233
162399790,5rtkit-daemon0-21swapper/323:37:323
162399790,5rtkit-daemon0-21swapper/323:37:313
162399780,5rtkit-daemon0-21swapper/323:20:143
162399780,5rtkit-daemon0-21swapper/322:01:453
162399780,5rtkit-daemon0-21swapper/321:33:443
162399770,5rtkit-daemon0-21swapper/321:51:063
162399770,5rtkit-daemon0-21swapper/321:51:053
162399770,5rtkit-daemon0-21swapper/319:24:203
162399770,5rtkit-daemon0-21swapper/300:08:563
25182042760,4chrt2517746-21/usr/sbin/munin23:10:201
162399750,25rtkit-daemon0-21swapper/320:45:413
162399750,25rtkit-daemon0-21swapper/320:45:413
162399730,7rtkit-daemon391irq_work/320:18:473
243039627245,17sleep10-21swapper/119:05:131
243149927146,18sleep20-21swapper/219:05:402
243197999671,37cyclictest2539325-21cpuspeed_turbos00:10:143
243151626739,22sleep30-21swapper/319:05:533
243197999652,38cyclictest1753-21gnome-shell00:15:253
243197999642,36cyclictest1753-21gnome-shell21:06:513
243197999642,36cyclictest1753-21gnome-shell19:31:203
243160126342,14sleep00-21swapper/019:06:570
243197999622,34cyclictest1753-21gnome-shell22:59:463
243197999622,34cyclictest1753-21gnome-shell22:59:453
243197999622,34cyclictest1753-21gnome-shell20:54:213
243197999622,32cyclictest1753-21gnome-shell20:20:283
25051122610,5chrt0-21swapper/022:35:130
243197999612,33cyclictest1753-21gnome-shell22:11:513
243197999612,33cyclictest1753-21gnome-shell21:27:243
243197999611,42cyclictest860997-21snmpd23:59:383
243197999611,33cyclictest2476434-21munin-run21:15:013
243197999602,33cyclictest1753-21gnome-shell19:30:003
243197999601,35cyclictest2449685-21taskset19:55:363
24411722590,9sleep01753-21gnome-shell19:35:010
243197999592,34cyclictest2495433-21seq22:09:423
243197999592,33cyclictest2523431-21/usr/sbin/munin23:25:223
243197999592,30cyclictest1753-21gnome-shell20:33:463
243197999582,33cyclictest860997-21snmpd21:44:163
243197999582,32cyclictest1753-21gnome-shell22:38:063
243197999582,31cyclictest2445857-21/usr/sbin/munin19:45:273
243197999581,34cyclictest2535735-21/usr/sbin/munin00:00:223
243197999581,33cyclictest860997-21snmpd23:54:223
243197999581,32cyclictest2531307-31apt-check23:45:303
243197999581,31cyclictest1753-21gnome-shell23:08:153
243197299583,50cyclictest2461473-21hddtemp_smartct20:30:221
162399580,8rtkit-daemon1622-21rtkit-daemon22:54:520
243197999572,30cyclictest1753-21gnome-shell19:35:193
243197999572,30cyclictest1753-21gnome-shell00:28:253
243197999562,30cyclictest1753-21gnome-shell20:26:593
243197999562,29cyclictest1753-21gnome-shell22:54:303
243197999551,31cyclictest860997-21snmpd23:00:483
24904642540,7sleep234-21ksoftirqd/221:55:032
24904642540,7sleep234-21ksoftirqd/221:55:022
243197999532,28cyclictest0-21swapper/300:31:143
2431979995121,26cyclictest2435732-21sshd19:18:273
2431979995121,26cyclictest2435732-21sshd19:18:263
243197999492,42cyclictest1753-21gnome-shell19:50:273
2431979994919,26cyclictest2433748-21cat19:15:013
162399490,34rtkit-daemon0-21swapper/223:07:212
243197999482,41cyclictest2478383-21/usr/sbin/munin21:20:193
2431979994823,20cyclictest1753-21gnome-shell00:35:193
243197999481,43cyclictest2504026-40apt-check22:30:273
243197999481,43cyclictest2476823-21cpuspeed_turbos21:15:153
243197999472,41cyclictest1753-21gnome-shell20:58:063
243197999472,40cyclictest1753-21gnome-shell23:13:143
243197999472,40cyclictest1753-21gnome-shell20:08:123
243197999471,42cyclictest1753-21gnome-shell00:23:543
243197999471,42cyclictest1753-21gnome-shell00:23:533
2431979994714,29cyclictest2499609-21expr22:20:143
2431979994712,31cyclictest1753-21gnome-shell23:32:383
2431979994712,31cyclictest1753-21gnome-shell23:32:383
243197999468,34cyclictest1753-21gnome-shell22:47:263
2431979994621,21cyclictest42-21ksoftirqd/320:36:013
2431979994616,25cyclictest2472007-21cat21:00:243
2431979994611,30cyclictest2454479-21grep20:10:273
2431970994624,17cyclictest2514226-21expr23:00:160
24351472450,15sleep2341ksoftirqd/219:15:272
24351472450,15sleep2341ksoftirqd/219:15:272
243197999459,32cyclictest1753-21gnome-shell20:04:443
24382782440,19chrt26-21ksoftirqd/119:25:201
2431979994414,25cyclictest2497559-21cpuspeed22:15:163
2431975994424,16cyclictest34-21ksoftirqd/222:17:412
2431975994421,13cyclictest34-21ksoftirqd/220:19:002
2431972994425,15cyclictest2483400-21latency_hist21:35:021
2431970994427,12cyclictest855-21dbus-daemon20:07:360
2431979994318,21cyclictest1069-21runrttasks22:28:533
2431979994318,21cyclictest1069-21runrttasks22:28:523
243197999431,37cyclictest2508272-21taskset22:42:203
2431972994320,19cyclictest26-21ksoftirqd/122:30:151
243197099432,16cyclictest2197014-21PK-Backend21:23:330
243197099431,16cyclictest2197014-21PK-Backend21:16:430
2431975994225,12cyclictest438-20systemd-journal19:10:282
2431975994223,14cyclictest34-21ksoftirqd/223:54:522
162399420,28rtkit-daemon2197014-21PK-Backend21:10:192
243197599417,13cyclictest34-21ksoftirqd/221:55:482
2431975994123,15cyclictest2437964-21/usr/sbin/munin19:25:152
2431975994122,15cyclictest3242255-21gdbus20:45:002
2431975994119,17cyclictest1-21systemd21:00:002
2431972994126,10cyclictest2499861-21/usr/sbin/munin22:20:201
243197299412,12cyclictest2535540-21latency_hist00:00:031
2431972994118,9cyclictest26-21ksoftirqd/122:29:561
2431972994118,9cyclictest26-21ksoftirqd/122:29:551
243197099412,7cyclictest2461449-21diskstats20:30:200
2431970994121,15cyclictest3242253-21nautilus23:35:030
2431970994121,15cyclictest3242253-21nautilus23:35:030
2431970994110,6cyclictest2520973-21turbostat23:20:010
2431975994022,14cyclictest34-21ksoftirqd/222:54:552
2431975994021,6cyclictest34-21ksoftirqd/219:45:012
2431975994021,14cyclictest34-21ksoftirqd/222:30:362
2431975994021,14cyclictest34-21ksoftirqd/219:38:212
2431972994025,11cyclictest2451766-21cron20:05:011
2431972994024,12cyclictest2433468-21seq19:12:571
2431972994023,12cyclictest438-20systemd-journal20:36:291
2431972994022,14cyclictest3242253-21nautilus23:45:011
2431972994017,19cyclictest2508561-21unin-run22:45:011
243197299401,35cyclictest2516806-21cut23:05:291
243197099406,6cyclictest141rcu_preempt23:30:000
2431970994027,8cyclictest1753-21gnome-shell23:10:070
2431970994026,10cyclictest2444836-21cron19:45:010
2431970994025,11cyclictest1753-21gnome-shell20:37:090
2431970994024,12cyclictest2494961-21cat22:05:330
2431970994024,12cyclictest2491844-21sshd21:58:080
2431970994024,12cyclictest2441853-21expr19:35:160
243197099402,33cyclictest1753-21gnome-shell20:25:190
2431970994017,6cyclictest141rcu_preempt20:20:280
243197099401,36cyclictest0-21swapper/000:07:530
25399582390,7sleep12197014-21PK-Backend00:10:261
2431975993923,12cyclictest2545898-21/usr/sbin/munin00:30:012
2431975993922,5cyclictest34-21ksoftirqd/223:03:052
2431975993921,6cyclictest34-21ksoftirqd/220:02:122
2431975993913,8cyclictest141rcu_preempt00:15:252
243197299396,6cyclictest141rcu_preempt21:35:241
243197299392,7cyclictest1753-21gnome-shell19:15:191
243197299392,7cyclictest1753-21gnome-shell19:15:181
2431972993924,11cyclictest2489236-21basename21:50:211
2431972993924,11cyclictest2489236-21basename21:50:211
243197299392,32cyclictest1753-21gnome-shell00:22:051
243197299392,32cyclictest1753-21gnome-shell00:22:051
2431972993922,13cyclictest26-21ksoftirqd/100:02:161
2431972993922,13cyclictest2496996-21sshd22:12:311
2431972993921,13cyclictest2453316-21systemd-udevd20:07:361
243197299391,34cyclictest231irq_work/120:20:211
2431972993911,12cyclictest2527860-40apt-check23:36:171
2431972993911,12cyclictest2527860-40apt-check23:36:171
2431970993924,11cyclictest2549576-21cut00:40:030
2431970993924,11cyclictest2454125-21cat20:10:180
243197099392,33cyclictest1753-21gnome-shell22:44:470
2431970993923,12cyclictest2449515-21cat19:55:300
2431970993922,8cyclictest1753-21gnome-shell22:00:310
2431970993922,13cyclictest2532999-21ntp_states23:50:270
2431970993920,14cyclictest13-21ksoftirqd/000:15:280
2431970993919,16cyclictest2521610-21/usr/sbin/munin23:20:160
2431970993918,17cyclictest1753-21gnome-shell23:30:400
2431970993918,17cyclictest1753-21gnome-shell23:30:390
24564392380,5sleep0161rcuc/020:15:300
243197599388,26cyclictest2446580-21latency_hist19:49:592
2431975993827,7cyclictest2474671-21apt21:10:012
2431975993821,13cyclictest34-21ksoftirqd/223:49:332
2431975993813,6cyclictest2527124-21kworker/u8:000:00:202
2431972993824,10cyclictest2520917-21sshd23:19:311
2431972993824,10cyclictest2514470-21interrupts23:00:221
2431972993823,11cyclictest2454112-21expr20:10:171
2431972993822,12cyclictest2525082-21diskstats23:30:161
2431972993822,12cyclictest2525082-21diskstats23:30:151
2431972993822,12cyclictest2456732-21sshd20:16:291
2431972993816,18cyclictest26-21ksoftirqd/123:20:201
2431972993812,22cyclictest2479327-21sed21:20:321
243197099386,6cyclictest141rcu_preempt20:41:420
243197099385,30cyclictest1069-21runrttasks21:46:440
243197099384,9cyclictest13-21ksoftirqd/023:05:020
243197099383,6cyclictest17531gnome-shell23:45:230
243197099382,32cyclictest1753-21gnome-shell21:33:100
2431970993823,11cyclictest2509093-21expr22:45:160
2431970993823,11cyclictest2437120-21cat19:20:320
2431970993822,12cyclictest2502426-21sshd22:26:320
2431970993822,12cyclictest2502426-21sshd22:26:310
2431970993813,21cyclictest2503910-40lsb_release22:30:230
2431970993812,5cyclictest141rcu_preempt20:00:260
243197099381,12cyclictest2470621-21chrt20:55:320
2431975993721,12cyclictest2529947-21apt-get23:45:012
243197599372,10cyclictest1753-21gnome-shell00:35:152
2431975993719,5cyclictest34-21ksoftirqd/221:35:152
2431975993719,14cyclictest34-21ksoftirqd/223:33:312
2431975993719,14cyclictest34-21ksoftirqd/223:33:312
2431975993719,14cyclictest34-21ksoftirqd/223:20:522
2431975993719,13cyclictest2287-21gmain23:35:052
2431975993719,13cyclictest2287-21gmain23:35:042
2431975993717,15cyclictest1-21systemd00:15:012
243197299373,7cyclictest141rcu_preempt21:15:031
243197299372,8cyclictest1753-21gnome-shell21:05:411
243197299372,31cyclictest1753-21gnome-shell20:00:021
2431972993723,10cyclictest26-21ksoftirqd/122:55:011
2431972993720,13cyclictest2490906-21df21:55:171
2431972993720,13cyclictest2468592-21memory20:50:241
2431972993719,13cyclictest2509163-21df22:45:181
2431972993717,15cyclictest3242253-21nautilus22:40:011
2431972993713,20cyclictest2444236-21seq19:40:301
243197099374,8cyclictest141rcu_preempt22:15:300
243197099374,5cyclictest141rcu_preempt19:25:580
2431970993726,7cyclictest2445793-21grep19:45:260
2431970993722,11cyclictest2486840-21systemd21:45:010
2431970993722,11cyclictest2481621-21turbostat21:30:000
2431970993721,12cyclictest2435651-21sshd19:17:270
2431970993721,12cyclictest2435651-21sshd19:17:260
2431970993720,13cyclictest2500627-40apt-check22:21:130
2431970993720,12cyclictest2475118-21grep21:10:160
2431970993719,13cyclictest2433882-21awk19:15:020
162399370,26rtkit-daemon2197014-21PK-Backend19:32:332
162399370,25rtkit-daemon0-21swapper/222:00:422
162399370,25rtkit-daemon0-21swapper/220:52:042
25066912360,7sleep22197014-21PK-Backend22:40:032
2431975993625,7cyclictest2449333-21sh19:55:272
2431975993622,10cyclictest34-21ksoftirqd/221:25:192
2431975993621,11cyclictest34-21ksoftirqd/220:21:152
243197599361,9cyclictest2197014-21PK-Backend21:30:052
243197599361,9cyclictest2197014-21PK-Backend20:47:182
243197599361,9cyclictest2197014-21PK-Backend20:47:172
243197599361,9cyclictest2197014-21PK-Backend19:20:032
2431975993619,13cyclictest2537863-21df_inode00:05:182
2431975993619,12cyclictest438-20systemd-journal22:15:002
2431975993617,14cyclictest1-21systemd23:15:012
243197599361,13cyclictest2197014-21PK-Backend22:05:212
243197299369,6cyclictest141rcu_preempt20:40:331
243197299368,7cyclictest141rcu_preempt19:24:591
243197299362,30cyclictest1753-21gnome-shell23:29:531
2431972993620,12cyclictest855-21dbus-daemon23:45:041
2431972993619,12cyclictest438-20systemd-journal20:30:011
2431972993617,15cyclictest2547622-21latency_hist00:35:021
2431972993616,17cyclictest2498982-21unin-run22:20:011
2431972993616,16cyclictest2492299-21cpu22:00:161
243197299361,31cyclictest2495477-21cut22:10:001
2431972993612,9cyclictest26-21ksoftirqd/100:27:251
243197299361,15cyclictest141rcu_preempt20:58:061
243197099366,10cyclictest141rcu_preempt21:05:530
243197099365,6cyclictest141rcu_preempt19:52:080
243197099365,5cyclictest141rcu_preempt23:45:010
243197099364,11cyclictest13-21ksoftirqd/000:26:340
2431970993620,12cyclictest2489591-21awk21:50:280
2431970993620,12cyclictest2489591-21awk21:50:270
2431970993620,12cyclictest2467145-21python320:45:300
2431970993620,12cyclictest2467145-21python320:45:300
2431970993620,11cyclictest2542970-21cat00:20:160
2431970993620,11cyclictest2542970-21cat00:20:150
2431970993619,13cyclictest2534564-21cat23:55:220
2431970993614,14cyclictest13-21ksoftirqd/020:52:340
243197599357,5cyclictest141rcu_preempt23:58:432
243197599356,9cyclictest34-21ksoftirqd/220:07:372
2431975993525,7cyclictest2472935-21grep21:05:002
2431975993518,13cyclictest1-21systemd21:25:012
243197599351,6cyclictest2197014-21PK-Backend22:40:472
243197299356,8cyclictest26-21ksoftirqd/121:00:261
2431972993524,7cyclictest2532421-21cpuspeed_turbos23:50:141
2431972993523,8cyclictest1753-21gnome-shell21:16:461
2431972993520,11cyclictest2442974-21sshd19:38:281
2431972993510,15cyclictest1753-21gnome-shell21:42:121
2431970993520,11cyclictest2512791-21grep22:55:240
2431970993520,11cyclictest2512791-21grep22:55:230
2431970993519,12cyclictest2547014-21awk00:30:300
243197099351,7cyclictest141rcu_preempt21:35:440
243197599349,5cyclictest141rcu_preempt21:44:112
243197599344,11cyclictest0-21swapper/220:10:222
2431975993419,11cyclictest2962348-21rs:main2
2431975993418,12cyclictest2543800-21seq00:20:392
2431975993418,12cyclictest2543800-21seq00:20:382
2431975993418,12cyclictest2447719-21python319:50:302
2431975993418,11cyclictest994-21lldpd22:59:002
2431975993418,11cyclictest994-21lldpd22:58:592
2431975993417,6cyclictest34-21ksoftirqd/222:25:072
2431975993417,6cyclictest34-21ksoftirqd/222:25:072
2431975993415,6cyclictest34-21ksoftirqd/223:16:002
2431972993421,10cyclictest2466249-21/usr/sbin/munin20:45:361
2431972993421,10cyclictest2466249-21/usr/sbin/munin20:45:351
2431972993419,11cyclictest2488147-21cat21:45:321
2431972993419,11cyclictest2440416-21grep19:30:301
2431972993418,11cyclictest2512855-21missed_timers22:55:241
2431972993418,11cyclictest2512855-21missed_timers22:55:241
2431972993415,15cyclictest2548262-21df_inode00:35:171
243197099344,5cyclictest141rcu_preempt22:11:560
243197099341,28cyclictest2179-21gvfsd-metadata00:05:010
2431975993317,12cyclictest2546231-21cat00:30:142
243197299334,8cyclictest1753-21gnome-shell00:10:001
2431972993321,8cyclictest1753-21gnome-shell00:16:061
2431972993311,18cyclictest2446107-21cat19:45:321
2431970993326,4cyclictest2197014-21PK-Backend21:03:220
162399330,16rtkit-daemon34-21ksoftirqd/220:38:112
24877432320,7chrt2197014-21PK-Backend21:45:242
2431975993216,12cyclictest2477733-21cat21:15:322
2431975993215,5cyclictest34-21ksoftirqd/222:21:112
243197599321,4cyclictest2197014-21PK-Backend22:45:172
2431975993210,18cyclictest1753-21gnome-shell20:30:422
243197299321,9cyclictest26-21ksoftirqd/119:50:321
2431972993217,11cyclictest2480333-21tr21:25:141
243197099326,10cyclictest141rcu_preempt00:13:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional