You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 17:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Dec 07, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
249920224520,11sleep00-21swapper/007:09:560
250638021100,18sleep342-21ksoftirqd/307:25:323
162399900,73rtkit-daemon0-21swapper/207:06:212
25705602880,7sleep02197014-21PK-Backend10:30:030
162399850,9rtkit-daemon1622-21rtkit-daemon08:11:320
162399820,8rtkit-daemon1622-21rtkit-daemon10:58:223
249907127646,21sleep30-21swapper/307:08:203
2499367997420,48cyclictest438-20systemd-journal09:35:001
26104812660,28sleep342-21ksoftirqd/312:25:013
26104812660,28sleep342-21ksoftirqd/312:25:013
162399660,9rtkit-daemon1622-21rtkit-daemon09:45:213
162399650,9rtkit-daemon1622-21rtkit-daemon10:29:313
162399640,9rtkit-daemon1622-21rtkit-daemon12:01:253
2499375996315,26cyclictest1753-21gnome-shell09:25:263
162399630,9rtkit-daemon1622-21rtkit-daemon10:22:583
162399630,10rtkit-daemon1622-21rtkit-daemon10:01:463
2499375996214,29cyclictest1753-21gnome-shell11:13:213
2499375996214,26cyclictest1753-21gnome-shell07:50:163
2499367996217,15cyclictest2613987-21gpgv12:35:011
162399620,9rtkit-daemon1622-21rtkit-daemon11:54:453
2499375996115,28cyclictest1753-21gnome-shell12:07:453
2499375996114,28cyclictest1753-21gnome-shell10:17:403
2499375996114,28cyclictest1753-21gnome-shell07:21:373
2499375996113,29cyclictest1753-21gnome-shell12:14:013
2499375996113,29cyclictest1753-21gnome-shell12:14:013
2499375996113,28cyclictest1753-21gnome-shell07:41:073
2499367996117,12cyclictest2552771-21sshd09:38:131
162399610,9rtkit-daemon1622-21rtkit-daemon11:26:273
162399610,9rtkit-daemon1622-21rtkit-daemon10:33:523
162399610,9rtkit-daemon1622-21rtkit-daemon10:09:223
249937599608,34cyclictest2529340-40lsb_release08:30:263
2499375996015,27cyclictest1753-21gnome-shell08:29:563
2499375996015,26cyclictest1753-21gnome-shell09:15:373
2499375996014,28cyclictest1753-21gnome-shell09:43:363
2499375996014,27cyclictest1753-21gnome-shell12:32:273
2499375996014,27cyclictest1753-21gnome-shell11:21:113
2499375996014,27cyclictest1753-21gnome-shell07:16:523
2499375996014,26cyclictest1753-21gnome-shell12:37:153
2499375996014,26cyclictest1753-21gnome-shell12:19:193
2499375996014,20cyclictest2575088-21/usr/sbin/munin10:40:283
2499375996013,29cyclictest1753-21gnome-shell10:39:593
2499375996013,28cyclictest1753-21gnome-shell07:58:303
2499367996019,9cyclictest3242253-21nautilus09:45:011
2499367996017,11cyclictest2506434-21sshd07:26:131
2499367996016,12cyclictest2602801-21uname12:00:281
249847326040,12sleep10-21swapper/107:05:231
162399600,8rtkit-daemon1622-21rtkit-daemon11:47:143
2499375995915,18cyclictest1753-21gnome-shell08:00:163
2499375995914,27cyclictest1753-21gnome-shell11:44:083
2499375995914,26cyclictest1753-21gnome-shell07:32:583
2499375995913,28cyclictest1753-21gnome-shell07:13:523
2499375995913,28cyclictest1753-21gnome-shell07:13:523
2499375995913,27cyclictest1753-21gnome-shell09:56:393
2499367995918,8cyclictest1069-21runrttasks07:20:251
2499367995916,12cyclictest884-21gdbus07:19:081
2499367995916,12cyclictest2569953-21sshd10:26:131
25570862580,6sleep20-21swapper/209:50:202
2499375995814,26cyclictest1753-21gnome-shell11:08:173
2499375995814,25cyclictest1753-21gnome-shell08:54:043
2499375995813,27cyclictest1753-21gnome-shell11:16:023
2499375995813,27cyclictest1753-21gnome-shell11:16:023
2499375995813,27cyclictest1753-21gnome-shell09:08:143
2499375995813,27cyclictest1753-21gnome-shell08:49:173
2499375995813,27cyclictest1753-21gnome-shell08:49:163
2499375995813,26cyclictest1753-21gnome-shell09:12:013
2499375995812,27cyclictest1753-21gnome-shell08:20:453
249936299581,12cyclictest1069-21runrttasks10:48:170
249936299581,11cyclictest2533501-21taskset08:44:490
162399580,8rtkit-daemon1622-21rtkit-daemon08:00:490
25240142570,20sleep342-21ksoftirqd/308:15:313
2499375995714,26cyclictest1753-21gnome-shell09:21:233
2499375995714,26cyclictest1753-21gnome-shell08:14:533
2499375995714,25cyclictest1753-21gnome-shell11:33:543
2499375995713,26cyclictest1753-21gnome-shell11:55:363
2499375995713,26cyclictest1753-21gnome-shell11:00:543
2499367995719,8cyclictest2540725-21/usr/sbin/munin09:05:181
2499367995717,11cyclictest2542896-21which09:10:201
2499367995716,13cyclictest2582020-21awk11:00:281
249936799571,12cyclictest2559848-21cron10:00:001
25572482560,7sleep00-21swapper/009:50:220
25464732560,14sleep0161rcuc/009:20:240
2499375995614,17cyclictest1753-21gnome-shell08:39:383
2499375995613,25cyclictest1-21systemd07:40:013
2499367995614,12cyclictest855-21dbus-daemon08:05:011
2499367995613,12cyclictest2534800-21cat08:45:301
2499367995613,12cyclictest2534800-21cat08:45:301
162399560,9rtkit-daemon1622-21rtkit-daemon11:53:012
249937599558,23cyclictest2552503-21sed09:35:313
2499375995514,24cyclictest1753-21gnome-shell10:10:433
2499375995513,25cyclictest1753-21gnome-shell09:04:493
2499375995513,25cyclictest1753-21gnome-shell07:48:553
249936799552,9cyclictest2580384-21python310:55:321
249936799552,7cyclictest2509931-21python307:35:291
2499367995520,7cyclictest2524740-21/usr/sbin/munin08:20:291
2499367995514,12cyclictest2600508-21cstates11:55:141
2499362995523,27cyclictest2550487-31apt-check09:30:300
249936299552,11cyclictest2592817-21taskset11:31:260
249936299552,10cyclictest2608567-21taskset12:17:500
2499362995519,31cyclictest2597083-21ls11:45:140
162399550,7rtkit-daemon1622-21rtkit-daemon07:58:420
2499375995414,16cyclictest1753-21gnome-shell08:55:163
2499367995416,10cyclictest2563777-21expr10:10:141
249936799541,12cyclictest2597253-21smartctl11:45:201
2499367995411,14cyclictest2548791-21awk09:25:341
2499375995314,14cyclictest2576915-21python310:45:333
249936799536,22cyclictest2528610-21apt-check08:30:131
2499367995320,8cyclictest26-21ksoftirqd/107:40:221
2499367995317,9cyclictest26-21ksoftirqd/111:25:021
2499362995320,29cyclictest2530933-21expr08:35:240
162399530,43rtkit-daemon2197014-21PK-Backend11:15:292
162399530,43rtkit-daemon2197014-21PK-Backend11:15:292
249937599529,18cyclictest1753-21gnome-shell12:27:243
2499375995214,14cyclictest2550305-21memory09:30:263
2499367995216,11cyclictest2582690-21cron11:09:551
2499367995214,13cyclictest2527810-21apt-get08:27:501
249936299521,46cyclictest1753-21gnome-shell12:00:510
162399520,8rtkit-daemon1622-21rtkit-daemon10:20:552
249937599518,18cyclictest1753-21gnome-shell08:05:183
249936299512,5cyclictest0-21swapper/010:17:230
2499362995120,27cyclictest855-21dbus-daemon09:00:010
2499362995118,28cyclictest2562208-21expr10:05:180
2499362995115,32cyclictest2586156-21cron11:15:000
2499362995110,37cyclictest1753-21gnome-shell12:06:530
2499362995110,37cyclictest1753-21gnome-shell09:14:150
249937599509,17cyclictest1753-21gnome-shell09:51:593
2499375995013,15cyclictest1753-21gnome-shell10:54:483
2499375995010,16cyclictest2533563-21munin-run08:45:013
249936799502,19cyclictest2507659-21df_inode07:30:171
2499367995019,6cyclictest2556492-21latency_hist09:50:021
2499367995017,9cyclictest64850irq/126-i91508:55:161
249936799501,26cyclictest908-21gdbus12:10:551
249936799501,26cyclictest908-21gdbus12:10:551
249936299508,38cyclictest1753-21gnome-shell09:38:060
249936299508,37cyclictest1753-21gnome-shell10:21:190
2499362995019,26cyclictest2610235-21seq12:22:000
2499362995019,26cyclictest2610235-21seq12:22:000
2499362995016,28cyclictest2586645-21cat11:15:150
2499362995016,28cyclictest2586645-21cat11:15:150
2499362995015,30cyclictest2536420-21swap08:50:310
249936299501,45cyclictest1753-21gnome-shell10:51:040
2499367994918,9cyclictest26-21ksoftirqd/107:10:161
2499367994918,9cyclictest26-21ksoftirqd/107:10:161
2499367994913,12cyclictest2560810-21sshd10:00:241
249936799491,22cyclictest2530310-21/usr/sbin/munin08:35:411
249936299499,35cyclictest1753-21gnome-shell07:35:090
2499362994920,24cyclictest2600684-21sh11:55:160
2499362994920,24cyclictest1753-21gnome-shell10:29:360
2499362994916,28cyclictest2527420-21grep08:25:330
2499362994914,30cyclictest438-20systemd-journal07:35:010
2499362994914,30cyclictest2545228-21cat09:15:300
2499362994913,31cyclictest1-21systemd11:30:010
2499362994911,34cyclictest2555177-21cat09:45:160
2499362994910,34cyclictest1753-21gnome-shell10:43:060
2499375994811,14cyclictest2593755-21df11:35:163
2499375994811,14cyclictest2593755-21df11:35:163
249936799488,10cyclictest2523679-21tr08:15:231
2499367994818,7cyclictest64850irq/126-i91510:10:021
2499367994815,12cyclictest1-21systemd09:01:561
249936299488,35cyclictest1753-21gnome-shell12:34:390
2499362994820,24cyclictest2534789-21timerandwakeup08:45:310
2499362994820,24cyclictest2534789-21timerandwakeup08:45:300
2499362994820,24cyclictest2511514-21head07:40:230
2499362994820,23cyclictest2514510-21cut07:50:140
2499362994816,27cyclictest2584074-21awk11:05:350
2499362994814,30cyclictest875-21polkitd07:19:080
2499362994812,31cyclictest875-21polkitd08:30:280
2499362994812,31cyclictest2614695-21cat12:35:140
249937099477,4cyclictest141rcu_preempt08:30:232
249936799472,8cyclictest2587185-21ntp_states11:15:261
249936799472,8cyclictest2587185-21ntp_states11:15:261
249936799471,15cyclictest2575392-21users10:40:321
249936299471,42cyclictest1753-21gnome-shell11:02:580
2499362994714,29cyclictest2589704-21sed11:25:010
2499362994713,29cyclictest438-20systemd-journal09:05:000
2499362994713,29cyclictest2523541-21latency08:15:220
249936299468,34cyclictest1753-21gnome-shell07:27:240
249936299468,33cyclictest2541180-21cat09:05:210
2499362994622,20cyclictest1753-21gnome-shell07:20:180
249936299461,41cyclictest1753-21gnome-shell12:14:480
249936299461,41cyclictest1753-21gnome-shell12:14:480
2499362994612,30cyclictest2560366-21cat10:00:150
2499362994612,29cyclictest2499778-21expr07:10:140
2499362994612,29cyclictest2499778-21expr07:10:140
2499362994611,31cyclictest2599592-21cat11:50:330
2499362994610,32cyclictest2553658-21if_err_enp2s009:40:190
2499362994610,31cyclictest2558615-21seq09:55:150
2499362994610,31cyclictest1753-21gnome-shell10:55:030
162399460,32rtkit-daemon1753-21gnome-shell07:43:362
25746892450,6chrt2197014-21PK-Backend10:40:192
25352422450,6chrt0-21swapper/208:49:552
25352422450,6chrt0-21swapper/208:49:552
249936299459,32cyclictest2573633-21timerwakeupswit10:35:320
249936299452,38cyclictest1753-21gnome-shell12:26:080
2499362994514,26cyclictest1069-21runrttasks09:25:030
162399450,33rtkit-daemon0-21swapper/208:16:392
249936299442,18cyclictest1753-21gnome-shell11:40:530
249936299432,37cyclictest1753-21gnome-shell11:36:230
249936299432,37cyclictest1753-21gnome-shell11:36:230
2499362994316,23cyclictest2197014-21PK-Backend08:22:320
2499362994314,24cyclictest2520000-21if_enp2s008:05:200
162399430,5rtkit-daemon0-21swapper/112:08:541
2499370994213,11cyclictest141rcu_preempt08:43:022
249936299422,35cyclictest2564621-21timerwakeupswit10:10:300
2499370994127,10cyclictest2584360-21sshd11:09:132
2499370994114,5cyclictest141rcu_preempt11:26:202
2499367994117,7cyclictest2514737-21colord-sane07:50:211
162399410,30rtkit-daemon2197014-21PK-Backend07:24:142
25852342400,5sleep2331rcuc/211:10:302
249937099407,10cyclictest141rcu_preempt12:31:562
249936799402,8cyclictest1753-21gnome-shell12:35:091
2499367994022,14cyclictest2566078-21ntp_states10:15:251
2499367994020,16cyclictest2532375-21cstates08:40:161
2499367994018,18cyclictest2516454-21df07:55:151
2499367994013,22cyclictest2546492-21cat09:20:241
249937099392,8cyclictest1753-21gnome-shell09:23:342
2499370993921,5cyclictest141rcu_preempt11:34:362
2499370993920,14cyclictest854-21avahi-daemon12:10:562
2499370993920,14cyclictest854-21avahi-daemon12:10:562
2499370993914,5cyclictest141rcu_preempt10:15:302
249936799391,10cyclictest0-21swapper/111:29:371
249937099382,7cyclictest1753-21gnome-shell08:24:092
2499370993826,8cyclictest1753-21gnome-shell07:18:292
2499370993822,12cyclictest2526906-21idleruntime08:25:242
2499370993821,12cyclictest438-20systemd-journal12:35:572
249937099381,8cyclictest2197014-21PK-Backend07:59:012
249937099381,33cyclictest1753-21gnome-shell09:12:192
249936799382,32cyclictest1753-21gnome-shell10:24:261
2499367993822,12cyclictest2570792-21apt10:30:131
2499367993819,14cyclictest438-20systemd-journal11:33:061
249936299382,9cyclictest1753-21gnome-shell07:45:290
162399380,26rtkit-daemon0-21swapper/109:50:591
249937099372,31cyclictest1753-21gnome-shell08:09:282
2499370993723,10cyclictest34-21ksoftirqd/211:20:132
2499370993723,10cyclictest2549189-21gpgv09:30:012
2499367993727,7cyclictest1069-21runrttasks08:53:031
2499367993720,7cyclictest2609231-21cstates12:20:151
2499367993720,7cyclictest2609231-21cstates12:20:151
2499367993710,22cyclictest2572772-21cat10:35:161
2499370993619,13cyclictest1-21systemd11:50:002
2499370993617,15cyclictest1753-21gnome-shell09:30:042
2499367993625,7cyclictest2544897-21strings09:15:231
2499367993611,10cyclictest2577889-21cpuspeed_turbos10:50:151
162399360,27rtkit-daemon1622-21rtkit-daemon10:06:082
162399360,25rtkit-daemon0-21swapper/111:35:221
162399360,25rtkit-daemon0-21swapper/111:35:211
2499370993524,7cyclictest2556382-21cron09:50:002
2499370993521,9cyclictest1753-21gnome-shell11:38:592
2499370993521,9cyclictest1753-21gnome-shell11:38:592
2499370993521,10cyclictest2595412-21expr11:40:142
2499370993520,11cyclictest2518186-21expr08:00:152
2499370993517,13cyclictest2570799-40apt-check10:30:182
2499370993513,5cyclictest141rcu_preempt07:50:332
2499370993512,8cyclictest141rcu_preempt08:37:392
2499370993511,4cyclictest141rcu_preempt10:45:132
2499367993519,12cyclictest1069-21runrttasks07:46:111
2499367993518,12cyclictest438-20systemd-journal12:30:001
249936799351,13cyclictest25214232tr08:10:161
249936799351,12cyclictest25764772/usr/sbin/munin10:45:231
26087722340,7sleep22197014-21PK-Backend12:20:002
249937099345,26cyclictest1069-21runrttasks08:13:182
249937099345,11cyclictest2197014-21PK-Backend09:39:042
2499370993421,10cyclictest34-21ksoftirqd/207:48:312
2499370993419,11cyclictest2539732-21ntp_kernel_pll_09:00:242
249937099341,6cyclictest1753-21gnome-shell07:33:482
2499370993416,14cyclictest2181-21gdbus10:30:022
2499370993413,5cyclictest141rcu_preempt07:30:012
249937099341,22cyclictest1753-21gnome-shell10:12:032
2499370993410,5cyclictest141rcu_preempt10:00:142
2499370993410,10cyclictest141rcu_preempt09:55:162
2499367993419,11cyclictest2598741-21grep11:50:161
2499367993418,12cyclictest2607270-21find12:15:121
249937099339,5cyclictest141rcu_preempt09:08:222
2499370993325,5cyclictest2197014-21PK-Backend12:04:192
2499370993319,10cyclictest2538760-21systemctl09:00:002
2499370993318,11cyclictest2573882-21sshd10:38:132
249937099331,28cyclictest2197014-21PK-Backend07:35:032
2499367993319,10cyclictest2519863-21expr08:05:141
249937099323,9cyclictest2197014-21PK-Backend11:00:312
2499370993222,7cyclictest1069-21runrttasks10:53:432
249937099321,9cyclictest2197014-21PK-Backend12:05:492
249937099321,5cyclictest2197014-21PK-Backend09:40:522
249937099321,4cyclictest2197014-21PK-Backend10:55:312
2499370993213,5cyclictest141rcu_preempt11:55:112
2499370993211,17cyclictest1753-21gnome-shell08:53:292
2499370993210,5cyclictest141rcu_preempt07:10:322
2499370993210,5cyclictest141rcu_preempt07:10:312
2499370993210,18cyclictest1753-21gnome-shell12:22:162
2499370993210,18cyclictest1753-21gnome-shell12:22:162
2499370993210,18cyclictest1753-21gnome-shell09:18:302
2499367993217,10cyclictest2595403-21cat11:40:141
26111382310,7chrt2197014-21PK-Backend12:25:202
2499367993113,8cyclictest881395-21snap11:11:521
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional