You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-01 - 13:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Apr 01, 2026 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
334937722460,9sleep20-21swapper/219:06:212
342643021230,3sleep00-21swapper/022:45:120
139099800,69rtkit-daemon0-21swapper/019:09:010
334818727346,16sleep10-21swapper/119:05:111
334860326541,16sleep30-21swapper/319:05:173
33689592480,3sleep00-21swapper/020:00:160
34479632460,3sleep20-21swapper/223:45:172
139099360,3rtkit-daemon3056-21PK-Backend20:30:291
334982299339,12cyclictest0-21swapper/222:58:222
34478352270,4sleep13056-21PK-Backend23:45:161
3349827992713,12cyclictest3387267-21unin-run20:55:013
139099270,18rtkit-daemon3056-21PK-Backend19:45:160
3349827992517,6cyclictest883-21dbus-daemon20:49:383
334982299252,7cyclictest34-21ksoftirqd/222:00:222
334981699256,7cyclictest13-21ksoftirqd/023:20:220
139099250,4rtkit-daemon1389-21rtkit-daemon23:57:572
3349822992412,10cyclictest3383136-21ps20:40:192
139099240,3rtkit-daemon1389-21rtkit-daemon23:53:222
334982299233,7cyclictest34-21ksoftirqd/222:45:012
3349822992315,6cyclictest3360889-21apt-get19:40:012
3349819992315,6cyclictest3359813-21fw_packets19:35:151
3349816992320,2cyclictest161rcuc/000:21:130
3349816992318,3cyclictest436-20systemd-journal23:16:590
34251782220,3sleep20-21swapper/222:40:172
34041152220,2sleep30-21swapper/321:40:193
3349827992220,1cyclictest0-21swapper/321:39:503
3349827992220,1cyclictest0-21swapper/320:31:433
334982799220,21cyclictest0-21swapper/322:51:153
334982299226,8cyclictest34-21ksoftirqd/200:22:202
334982299225,9cyclictest34-21ksoftirqd/220:16:452
334982299223,7cyclictest34-21ksoftirqd/221:39:142
334982299223,7cyclictest34-21ksoftirqd/220:28:002
334982299223,6cyclictest34-21ksoftirqd/200:13:312
334982299223,6cyclictest34-21ksoftirqd/200:13:312
3349822992220,1cyclictest0-21swapper/223:30:012
3349822992220,1cyclictest0-21swapper/223:26:442
3349822992220,1cyclictest0-21swapper/219:18:152
3349822992216,5cyclictest3395932-21systemd-run21:20:002
3349822992213,7cyclictest3386129-21df20:50:122
3349819992219,2cyclictest3056-21PK-Backend00:29:181
334981999220,20cyclictest3056-21PK-Backend21:35:201
334981699225,6cyclictest13-21ksoftirqd/021:00:010
334981699222,8cyclictest13-21ksoftirqd/019:35:180
3349816992220,1cyclictest0-21swapper/023:59:500
3349816992220,1cyclictest0-21swapper/023:42:130
3349816992220,1cyclictest0-21swapper/022:56:090
3349816992220,1cyclictest0-21swapper/022:38:250
3349816992220,1cyclictest0-21swapper/021:55:190
3349816992220,1cyclictest0-21swapper/021:52:150
3349816992220,1cyclictest0-21swapper/020:07:240
3349816992220,1cyclictest0-21swapper/019:33:590
3349816992219,2cyclictest3056-21PK-Backend20:40:360
3349816992218,3cyclictest161rcuc/020:25:400
3349816992218,3cyclictest13-21ksoftirqd/020:24:020
334981699220,9cyclictest13-21ksoftirqd/000:10:180
334981699220,9cyclictest13-21ksoftirqd/000:10:170
334981699220,6cyclictest13-21ksoftirqd/019:10:210
334981699220,21cyclictest0-21swapper/022:02:150
139099220,3rtkit-daemon1389-21rtkit-daemon23:41:112
139099220,15rtkit-daemon0-21swapper/021:29:570
33722102210,3sleep20-21swapper/220:10:132
334982799217,12cyclictest3442267-21cstates23:30:123
3349827992119,1cyclictest0-21swapper/322:48:593
3349827992119,1cyclictest0-21swapper/322:42:233
3349827992119,1cyclictest0-21swapper/322:31:413
3349827992119,1cyclictest0-21swapper/322:22:153
3349827992119,1cyclictest0-21swapper/322:10:323
3349827992119,1cyclictest0-21swapper/322:06:133
3349827992119,1cyclictest0-21swapper/321:32:403
3349827992119,1cyclictest0-21swapper/319:46:143
3349827992119,1cyclictest0-21swapper/300:32:553
3349827992113,7cyclictest883-21dbus-daemon20:25:003
334982799211,17cyclictest3387506-21apt20:55:113
334982799210,20cyclictest0-21swapper/323:41:103
334982799210,20cyclictest0-21swapper/300:37:483
334982799210,20cyclictest0-21swapper/300:00:513
334982799210,19cyclictest0-21swapper/321:10:213
334982299214,10cyclictest3465177-21taskset00:39:262
334982299213,7cyclictest34-21ksoftirqd/220:04:262
3349822992119,1cyclictest0-21swapper/222:05:522
3349822992119,1cyclictest0-21swapper/221:53:212
3349822992119,1cyclictest0-21swapper/221:11:442
3349822992119,1cyclictest0-21swapper/221:05:152
3349822992119,1cyclictest0-21swapper/220:38:192
3349822992119,1cyclictest0-21swapper/200:32:472
3349822992118,2cyclictest331rcuc/222:31:502
334982299211,7cyclictest34-21ksoftirqd/200:06:052
3349822992115,4cyclictest3433065-21nm-dispatcher23:04:412
334982299210,20cyclictest0-21swapper/221:00:362
334981999216,7cyclictest918-21rs:main1
3349819992119,1cyclictest3056-21PK-Backend21:26:481
3349819992119,1cyclictest0-21swapper/119:44:201
3349819992118,2cyclictest3056-21PK-Backend22:05:221
3349819992118,2cyclictest3056-21PK-Backend19:17:101
334981999210,7cyclictest3056-21PK-Backend20:14:591
334981999210,20cyclictest3056-21PK-Backend22:44:551
334981999210,20cyclictest3056-21PK-Backend21:23:571
334981999210,20cyclictest3056-21PK-Backend21:03:081
334981999210,1cyclictest3056-21PK-Backend22:47:591
334981999210,19cyclictest3056-21PK-Backend19:23:531
334981699212,6cyclictest13-21ksoftirqd/021:02:160
3349816992119,1cyclictest0-21swapper/023:06:030
3349816992119,1cyclictest0-21swapper/022:24:390
3349816992119,1cyclictest0-21swapper/022:12:570
3349816992119,1cyclictest0-21swapper/022:09:390
3349816992119,1cyclictest0-21swapper/021:16:400
3349816992119,1cyclictest0-21swapper/020:14:340
3349816992119,1cyclictest0-21swapper/019:20:330
3349816992119,1cyclictest0-21swapper/000:25:120
3349816992117,3cyclictest161rcuc/021:07:490
334981699210,20cyclictest0-21swapper/023:32:460
334981699210,20cyclictest0-21swapper/022:40:420
334981699210,20cyclictest0-21swapper/020:49:420
34403122200,4chrt3056-21PK-Backend23:25:111
3349827992018,1cyclictest0-21swapper/323:13:533
3349827992013,5cyclictest436-20systemd-journal21:09:543
3349827992012,6cyclictest3447529-21expr23:45:133
334982799200,8cyclictest3398765-21python321:25:203
334982799200,1cyclictest3056-21PK-Backend00:22:063
334982799200,1cyclictest0-21swapper/321:02:323
334982799200,19cyclictest0-21swapper/321:53:563
334982799200,19cyclictest0-21swapper/321:24:303
334982799200,19cyclictest0-21swapper/321:15:303
334982799200,19cyclictest0-21swapper/319:56:153
334982799200,19cyclictest0-21swapper/319:28:433
334982799200,19cyclictest0-21swapper/319:21:403
334982799200,19cyclictest0-21swapper/300:12:063
334982799200,19cyclictest0-21swapper/300:12:063
334982799200,18cyclictest3452228-40mandb00:00:013
334982299203,6cyclictest34-21ksoftirqd/222:50:312
334982299202,7cyclictest34-21ksoftirqd/221:42:112
334982299202,7cyclictest34-21ksoftirqd/219:25:372
334982299202,6cyclictest34-21ksoftirqd/221:48:322
334982299202,16cyclictest3444513-21tr23:35:172
3349822992018,1cyclictest0-21swapper/223:22:242
3349822992018,1cyclictest0-21swapper/222:19:032
3349822992018,1cyclictest0-21swapper/222:12:282
3349822992018,1cyclictest0-21swapper/221:29:232
3349822992018,1cyclictest0-21swapper/220:23:522
3349822992018,1cyclictest0-21swapper/219:51:272
3349822992017,2cyclictest3418228-21ntp_states22:20:182
3349822992017,2cyclictest331rcuc/221:31:362
3349822992014,5cyclictest3423275-21load22:35:162
3349822992014,4cyclictest3379495-21strings20:30:182
3349822992011,7cyclictest3453935-21apt-get00:05:002
334982299200,1cyclictest0-21swapper/222:26:412
334982299200,1cyclictest0-21swapper/221:59:312
334982299200,1cyclictest0-21swapper/219:45:252
334982299200,19cyclictest0-21swapper/220:59:352
334982299200,19cyclictest0-21swapper/219:57:292
334982299200,19cyclictest0-21swapper/219:11:202
334982299200,19cyclictest0-21swapper/200:29:432
3349819992018,1cyclictest3056-21PK-Backend23:57:141
3349819992018,1cyclictest3056-21PK-Backend21:19:171
3349819992018,1cyclictest3056-21PK-Backend21:11:481
3349819992018,1cyclictest3056-21PK-Backend21:08:231
3349819992018,1cyclictest3056-21PK-Backend00:19:451
3349819992017,2cyclictest3056-21PK-Backend20:43:151
334981999200,1cyclictest3056-21PK-Backend00:32:571
334981999200,19cyclictest3056-21PK-Backend00:02:081
334981999200,19cyclictest0-21swapper/123:00:581
334981999200,19cyclictest0-21swapper/120:22:391
334981999200,18cyclictest3056-21PK-Backend22:13:181
334981999200,18cyclictest3056-21PK-Backend19:58:061
3349816992018,1cyclictest3056-21PK-Backend00:05:110
3349816992018,1cyclictest0-21swapper/022:52:510
3349816992018,1cyclictest0-21swapper/022:17:160
3349816992018,1cyclictest0-21swapper/021:20:450
3349816992018,1cyclictest0-21swapper/020:50:440
3349816992018,1cyclictest0-21swapper/000:33:570
3349816992017,2cyclictest161rcuc/019:40:020
334981699200,1cyclictest0-21swapper/023:02:230
334981699200,1cyclictest0-21swapper/022:26:400
334981699200,1cyclictest0-21swapper/021:48:100
334981699200,1cyclictest0-21swapper/021:11:320
334981699200,1cyclictest0-21swapper/019:19:110
334981699200,19cyclictest0-21swapper/019:52:230
334981699200,19cyclictest0-21swapper/000:16:130
139099200,3rtkit-daemon1389-21rtkit-daemon23:13:132
139099200,2rtkit-daemon1389-21rtkit-daemon23:38:403
139099200,14rtkit-daemon3056-21PK-Backend20:39:251
3349827991917,1cyclictest0-21swapper/323:28:403
3349827991917,1cyclictest0-21swapper/322:00:053
3349827991917,1cyclictest0-21swapper/320:37:313
3349827991917,1cyclictest0-21swapper/320:15:343
3349827991917,1cyclictest0-21swapper/319:50:263
3349827991914,3cyclictest3350374-21diskstats19:10:143
3349827991912,5cyclictest3433721-21expr23:05:123
3349827991911,6cyclictest3352164-21if_enp2s019:15:023
334982799190,7cyclictest3457911-21users00:15:233
334982799190,1cyclictest3056-21PK-Backend22:55:343
334982799190,1cyclictest0-21swapper/323:20:153
334982799190,1cyclictest0-21swapper/320:06:013
334982799190,18cyclictest0-21swapper/321:57:003
334982799190,18cyclictest0-21swapper/300:27:003
334982799190,17cyclictest3383885-21sed20:45:013
334982299195,12cyclictest3438176-21python323:18:212
334982299193,10cyclictest34-21ksoftirqd/219:41:002
3349822991917,1cyclictest0-21swapper/220:49:272
3349822991917,1cyclictest0-21swapper/219:23:292
334982299190,18cyclictest0-21swapper/223:06:562
3349819991917,1cyclictest3056-21PK-Backend23:53:221
3349819991917,1cyclictest3056-21PK-Backend20:56:051
3349819991917,1cyclictest3056-21PK-Backend20:45:511
3349819991917,1cyclictest3056-21PK-Backend19:49:551
3349819991917,1cyclictest3056-21PK-Backend00:07:501
3349819991916,2cyclictest3056-21PK-Backend22:28:121
3349819991911,6cyclictest3438318-21cron23:20:001
3349819991911,6cyclictest3378551-21cron20:30:001
334981999190,1cyclictest3056-21PK-Backend23:41:461
334981999190,1cyclictest3056-21PK-Backend22:52:051
334981999190,1cyclictest3056-21PK-Backend21:48:311
334981999190,1cyclictest3056-21PK-Backend00:11:271
334981999190,1cyclictest3056-21PK-Backend00:11:271
334981999190,18cyclictest3056-21PK-Backend23:33:011
334981699192,4cyclictest13-21ksoftirqd/021:40:000
3349816991917,1cyclictest0-21swapper/020:34:210
3349816991917,1cyclictest0-21swapper/000:36:000
3349816991915,3cyclictest161rcuc/019:29:180
3349816991912,5cyclictest3453353-21sshd00:00:200
334981699190,1cyclictest0-21swapper/020:39:350
334982799183,13cyclictest3420714-21munin-plugin-st22:30:003
334982799182,15cyclictest0-21swapper/323:04:123
334982799182,14cyclictest3377070-21/usr/sbin/munin20:25:173
3349827991816,1cyclictest0-21swapper/323:18:133
334982799181,3cyclictest3372727-21python320:10:203
334982299184,3cyclictest34-21ksoftirqd/219:30:172
3349822991814,3cyclictest978-21snmpd00:19:242
3349822991811,5cyclictest3396324-21grep21:20:122
3349819991816,1cyclictest3056-21PK-Backend23:23:141
3349819991816,1cyclictest3056-21PK-Backend23:10:231
3349819991816,1cyclictest3056-21PK-Backend22:37:591
3349819991814,3cyclictest3359158-21munin-plugin-st19:35:001
3349819991811,5cyclictest3404962-21gpgv21:45:011
334981699186,2cyclictest141rcu_preempt20:16:590
3349816991815,2cyclictest161rcuc/021:32:350
3349816991813,3cyclictest3449125-21tr23:50:110
334981699181,16cyclictest3421158-21cpuspeed_turbos22:30:110
3349816991811,5cyclictest3435560-21awk23:10:160
3349816991810,6cyclictest3444652-21python323:35:190
334981699180,5cyclictest1788620-21WebKitWebProces23:25:240
3349827991714,2cyclictest3056-21PK-Backend19:35:053
3349827991712,4cyclictest3358682-21sed19:30:233
3349827991711,5cyclictest3450035-21sed23:50:203
334982799170,4cyclictest0-21swapper/322:37:203
334981999179,6cyclictest3357415-21perf19:30:001
3349819991715,1cyclictest3056-21PK-Backend00:35:471
334981699175,6cyclictest13-21ksoftirqd/019:55:260
334982799166,9cyclictest3453935-21apt-get00:05:013
3349827991614,1cyclictest0-21swapper/321:46:003
3349827991612,3cyclictest42-21ksoftirqd/320:01:033
3349827991610,4cyclictest3416407-21grep22:15:163
334982299168,6cyclictest3370343-21cat20:05:142
334981999169,5cyclictest3458687-21wc00:20:111
3349819991614,1cyclictest3056-21PK-Backend22:23:191
3349819991614,1cyclictest3056-21PK-Backend20:16:321
3349819991613,2cyclictest3056-21PK-Backend23:05:171
3349819991612,2cyclictest26-21ksoftirqd/119:55:011
3349819991611,3cyclictest3412019-21apt-key22:05:001
3349819991610,5cyclictest3350985-21cat19:10:221
334981999160,15cyclictest3056-21PK-Backend23:35:051
334981699162,3cyclictest13-21ksoftirqd/023:45:120
334981699162,3cyclictest13-21ksoftirqd/021:41:380
334982799159,4cyclictest3361409-21cat19:40:113
3349819991513,1cyclictest3056-21PK-Backend22:17:241
3349819991513,1cyclictest3056-21PK-Backend21:30:401
3349819991513,1cyclictest3056-21PK-Backend20:52:411
334981999150,14cyclictest3056-21PK-Backend20:02:521
139099150,3rtkit-daemon3056-21PK-Backend21:54:581
334981999140,13cyclictest3056-21PK-Backend22:33:061
34090562130,4sleep13056-21PK-Backend21:55:171
139099130,7rtkit-daemon3056-21PK-Backend20:06:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional