You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 03:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Apr 07, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
253597521510,26sleep10-21swapper/119:05:241
253552821060,5sleep32535377-21cstates19:05:153
253629027152,13sleep00-21swapper/019:05:530
253643026444,14sleep20-21swapper/219:07:392
26004402470,2sleep30-21swapper/322:10:023
253676499323,27cyclictest2579090-21awk21:10:001
253676499312,16cyclictest0-21swapper/120:15:421
2536764992911,16cyclictest2541558-21grep19:20:181
253676499290,3cyclictest436-20systemd-journal00:30:001
2536764992813,13cyclictest2539048-21sed19:15:031
253676499281,25cyclictest2558735-21cstates20:10:121
25642172270,8sleep342-21ksoftirqd/320:25:153
139099270,4rtkit-daemon1389-21rtkit-daemon20:18:310
253676499267,17cyclictest1-21systemd00:15:011
253676499261,23cyclictest2570265-21chrt20:43:211
253676499260,2cyclictest2549860-21apt19:45:101
253676999250,5cyclictest2604532-21chrt22:20:153
253676599250,23cyclictest2612048-21ntpq22:40:172
2536764992518,5cyclictest2639551-21grep00:00:121
253676499250,3cyclictest2567949-21uname20:35:181
139099250,4rtkit-daemon1389-21rtkit-daemon22:49:321
253676499248,14cyclictest2627916-21ntpq23:25:181
253676499247,15cyclictest2652683-21sed00:35:201
253676499245,17cyclictest2547628-21systemctl19:40:001
253676499242,18cyclictest2547868-21apt19:40:111
2536764992419,4cyclictest0-21swapper/120:25:451
2536764992412,10cyclictest2561971-21/usr/sbin/munin20:20:161
253676499241,18cyclictest2618277-21apt23:00:121
2536764992410,12cyclictest2567017-21wc20:35:011
26189172230,2sleep33056-21PK-Backend23:00:173
253676999230,21cyclictest3056-21PK-Backend19:22:243
2536765992315,5cyclictest436-20systemd-journal22:20:212
253676599230,22cyclictest0-21swapper/220:03:142
253676599230,21cyclictest2560259-21apt20:15:122
253676499236,16cyclictest2626466-21cat23:20:201
253676499236,15cyclictest2642697-21latency_hist00:10:011
253676499236,15cyclictest2615390-21missed_timers22:50:181
253676499236,14cyclictest2557002-21cstates20:05:131
253676499235,15cyclictest2553216-21latency_hist19:55:011
253676499232,17cyclictest0-21swapper/123:50:111
2536764992314,6cyclictest436-20systemd-journal23:19:591
2536764992312,9cyclictest2584106-21timerwakeupswit21:20:241
2536759992318,3cyclictest2627945-21ntp_states23:25:190
2536759992317,4cyclictest2547610-21basename19:40:010
139099230,3rtkit-daemon1389-21rtkit-daemon22:55:031
25834462220,3sleep20-21swapper/221:20:172
2536769992219,2cyclictest3056-21PK-Backend20:38:563
253676999220,20cyclictest3056-21PK-Backend23:48:373
253676599229,11cyclictest34-21ksoftirqd/200:14:552
2536765992220,1cyclictest0-21swapper/223:53:102
2536765992215,6cyclictest2549335-21apt-get19:45:002
2536765992214,6cyclictest2173-21JS10:27:482
253676499227,13cyclictest2644694-21apt-key00:15:011
253676499227,13cyclictest0-21swapper/123:55:101
253676499225,15cyclictest2631584-21switchtime23:35:211
253676499225,15cyclictest2620936-21needreboot23:05:171
2536764992220,1cyclictest0-21swapper/120:02:061
253676499220,4cyclictest436-20systemd-journal20:00:011
253676499220,3cyclictest2537350-21hddtemp_smartct19:10:131
253676499220,20cyclictest3056-21PK-Backend22:29:271
253676499220,20cyclictest0-21swapper/121:15:121
2536759992220,1cyclictest0-21swapper/023:37:560
2536759992220,1cyclictest0-21swapper/020:43:170
2536759992220,1cyclictest0-21swapper/000:20:280
2536759992220,1cyclictest0-21swapper/000:20:280
2536759992220,1cyclictest0-21swapper/000:14:330
2536759992218,3cyclictest2627011-21kworker/0:223:30:120
2536759992217,3cyclictest1788620-21WebKitWebProces20:35:060
2536759992213,7cyclictest2548728-21python319:40:190
2536759992213,7cyclictest2013-21gdbus22:12:180
253675999220,20cyclictest2579716-21diskstats21:10:140
2536769992119,1cyclictest3056-21PK-Backend23:56:213
2536769992119,1cyclictest3056-21PK-Backend20:16:053
2536769992119,1cyclictest0-21swapper/322:47:193
2536769992119,1cyclictest0-21swapper/322:32:533
2536769992119,1cyclictest0-21swapper/321:40:143
2536769992119,1cyclictest0-21swapper/321:02:223
253676999210,20cyclictest3056-21PK-Backend20:33:563
253676999210,20cyclictest0-21swapper/321:56:363
253676999210,19cyclictest3056-21PK-Backend00:37:373
2536765992119,1cyclictest0-21swapper/222:38:282
2536765992119,1cyclictest0-21swapper/222:31:592
2536765992119,1cyclictest0-21swapper/222:00:562
2536765992119,1cyclictest0-21swapper/221:59:072
2536765992119,1cyclictest0-21swapper/221:45:152
2536765992119,1cyclictest0-21swapper/221:01:342
2536765992119,1cyclictest0-21swapper/220:45:392
2536765992119,1cyclictest0-21swapper/220:43:162
253676599210,20cyclictest0-21swapper/222:49:222
253676599210,20cyclictest0-21swapper/221:09:322
253676499218,11cyclictest2632523-21cpuspeed_turbos23:40:131
253676499218,11cyclictest2629153-21cstates23:30:131
253676499218,11cyclictest2603880-21latency_hist22:20:011
253676499218,11cyclictest2603880-21latency_hist22:20:001
253676499217,12cyclictest2579752-21idleruntime21:10:151
253676499215,14cyclictest2571068-21fw_packets20:45:151
253676499211,9cyclictest2545877-21munin-run19:35:001
2536764992119,1cyclictest3056-21PK-Backend22:30:291
2536764992119,1cyclictest3056-21PK-Backend21:59:331
2536764992119,1cyclictest3056-21PK-Backend21:40:471
2536764992119,1cyclictest0-21swapper/100:20:331
2536764992119,1cyclictest0-21swapper/100:20:331
2536764992118,2cyclictest3056-21PK-Backend22:07:371
2536764992118,2cyclictest3056-21PK-Backend21:45:131
2536764992118,2cyclictest3056-21PK-Backend21:36:211
253676499210,19cyclictest3056-21PK-Backend23:13:221
253676499210,19cyclictest3056-21PK-Backend22:12:031
2536759992119,1cyclictest0-21swapper/023:50:350
2536759992119,1cyclictest0-21swapper/023:18:230
2536759992119,1cyclictest0-21swapper/022:25:040
2536759992119,1cyclictest0-21swapper/021:15:270
2536759992119,1cyclictest0-21swapper/019:48:550
2536759992114,6cyclictest2586261-21cron21:30:000
2536759992113,6cyclictest436-20systemd-journal21:10:010
253675999210,20cyclictest0-21swapper/023:00:040
253675999210,20cyclictest0-21swapper/022:58:010
253675999210,20cyclictest0-21swapper/022:30:300
253675999210,20cyclictest0-21swapper/020:33:160
253675999210,19cyclictest0-21swapper/019:10:080
139099210,17rtkit-daemon1788643-21eadedCompositor20:56:550
139099210,16rtkit-daemon2536753-21cyclictest19:57:380
2536769992018,1cyclictest3056-21PK-Backend23:28:153
2536769992018,1cyclictest3056-21PK-Backend19:54:013
2536769992018,1cyclictest3056-21PK-Backend19:39:353
2536769992018,1cyclictest3056-21PK-Backend00:25:143
2536769992018,1cyclictest3056-21PK-Backend00:05:543
2536769992018,1cyclictest0-21swapper/321:32:573
2536769992018,1cyclictest0-21swapper/320:59:453
2536769992018,1cyclictest0-21swapper/320:45:333
2536769992018,1cyclictest0-21swapper/319:28:463
253676999201,7cyclictest2539180-21/usr/sbin/munin19:15:113
2536769992017,2cyclictest3056-21PK-Backend23:30:183
2536769992017,2cyclictest3056-21PK-Backend22:55:523
2536769992017,2cyclictest3056-21PK-Backend00:12:363
253676999200,1cyclictest3056-21PK-Backend23:22:213
253676999200,1cyclictest3056-21PK-Backend22:50:103
253676999200,1cyclictest3056-21PK-Backend20:20:173
253676999200,1cyclictest3056-21PK-Backend19:14:553
253676999200,1cyclictest0-21swapper/322:03:533
253676999200,19cyclictest3056-21PK-Backend19:56:383
253676999200,19cyclictest0-21swapper/323:15:243
253676999200,19cyclictest0-21swapper/322:43:273
253676999200,19cyclictest0-21swapper/322:36:453
253676999200,19cyclictest0-21swapper/321:46:293
2536765992018,1cyclictest0-21swapper/223:56:482
2536765992018,1cyclictest0-21swapper/223:43:032
2536765992018,1cyclictest0-21swapper/223:10:252
2536765992018,1cyclictest0-21swapper/222:17:252
2536765992018,1cyclictest0-21swapper/222:17:252
2536765992018,1cyclictest0-21swapper/221:50:282
2536765992018,1cyclictest0-21swapper/221:30:342
2536765992018,1cyclictest0-21swapper/220:36:202
2536765992018,1cyclictest0-21swapper/219:50:312
2536765992015,3cyclictest2574503-21df20:55:122
2536765992012,6cyclictest2585154-21cat21:25:132
253676599200,19cyclictest0-21swapper/223:38:232
253676599200,19cyclictest0-21swapper/222:06:102
253676599200,19cyclictest0-21swapper/221:40:002
253676499203,15cyclictest0-21swapper/119:26:481
253676499203,15cyclictest0-21swapper/100:30:111
2536764992018,1cyclictest3056-21PK-Backend22:21:571
2536764992017,2cyclictest3056-21PK-Backend22:40:021
253676499200,19cyclictest3056-21PK-Backend22:01:221
253676499200,19cyclictest3056-21PK-Backend21:32:441
2536759992018,1cyclictest0-21swapper/023:57:160
2536759992018,1cyclictest0-21swapper/022:46:530
2536759992018,1cyclictest0-21swapper/022:40:370
2536759992018,1cyclictest0-21swapper/022:16:100
2536759992018,1cyclictest0-21swapper/022:16:100
2536759992018,1cyclictest0-21swapper/020:23:490
2536759992018,1cyclictest0-21swapper/020:04:230
2536759992018,1cyclictest0-21swapper/019:24:070
2536759992018,1cyclictest0-21swapper/000:19:270
2536759992018,1cyclictest0-21swapper/000:06:490
2536759992012,6cyclictest2540114-21threads19:15:200
253675999200,19cyclictest0-21swapper/020:53:030
253675999200,19cyclictest0-21swapper/020:27:140
253675999200,19cyclictest0-21swapper/000:02:570
2536769991917,1cyclictest3056-21PK-Backend20:03:273
2536769991917,1cyclictest0-21swapper/322:12:113
2536769991917,1cyclictest0-21swapper/321:36:363
2536769991916,2cyclictest3056-21PK-Backend23:35:123
2536769991916,2cyclictest3056-21PK-Backend20:11:533
2536769991916,2cyclictest3056-21PK-Backend00:31:563
2536769991914,3cyclictest2622374-21diskstats23:10:123
2536769991913,4cyclictest2605875-21apt22:25:123
2536769991912,5cyclictest2585191-21awk21:25:153
253676999190,18cyclictest0-21swapper/321:21:143
2536765991917,1cyclictest0-21swapper/223:32:572
2536765991917,1cyclictest0-21swapper/222:59:162
2536765991917,1cyclictest0-21swapper/222:11:252
2536765991917,1cyclictest0-21swapper/219:16:242
2536765991917,1cyclictest0-21swapper/200:23:052
2536765991917,1cyclictest0-21swapper/200:23:052
2536765991917,1cyclictest0-21swapper/200:06:422
2536765991917,1cyclictest0-21swapper/200:00:282
2536765991915,3cyclictest978-21snmpd19:25:252
2536765991915,3cyclictest34-21ksoftirqd/220:22:192
2536765991912,5cyclictest2542394-21basename19:24:592
253676599190,1cyclictest3056-21PK-Backend19:10:082
253676599190,1cyclictest0-21swapper/223:07:352
253676599190,1cyclictest0-21swapper/220:10:242
253676599190,1cyclictest0-21swapper/200:39:412
253676599190,1cyclictest0-21swapper/200:33:122
253676599190,18cyclictest0-21swapper/220:27:212
253676599190,18cyclictest0-21swapper/219:39:082
253676499190,7cyclictest2634352-21cstates23:45:111
253676499190,7cyclictest2575867-21awk21:00:131
253676499190,1cyclictest3056-21PK-Backend21:51:291
253676499190,1cyclictest3056-21PK-Backend21:28:181
253676499190,18cyclictest3056-21PK-Backend22:36:101
2536759991917,1cyclictest0-21swapper/023:46:420
2536759991917,1cyclictest0-21swapper/023:42:500
2536759991917,1cyclictest0-21swapper/022:35:100
2536759991917,1cyclictest0-21swapper/021:47:240
2536759991917,1cyclictest0-21swapper/019:31:180
2536759991917,1cyclictest0-21swapper/000:32:040
2536759991913,4cyclictest1095-21runrttasks22:50:170
2536759991911,6cyclictest436-20systemd-journal00:37:310
253675999190,1cyclictest0-21swapper/023:07:480
253675999190,1cyclictest0-21swapper/022:22:120
253675999190,1cyclictest0-21swapper/000:27:110
253675999190,18cyclictest0-21swapper/022:09:070
253675999190,18cyclictest0-21swapper/021:33:320
253675999190,18cyclictest0-21swapper/020:12:490
2536769991816,1cyclictest3056-21PK-Backend21:18:373
2536769991816,1cyclictest3056-21PK-Backend20:08:273
2536769991816,1cyclictest3056-21PK-Backend19:34:343
2536769991816,1cyclictest3056-21PK-Backend00:03:033
2536769991816,1cyclictest0-21swapper/319:43:003
2536769991815,2cyclictest3056-21PK-Backend23:05:383
253676999180,16cyclictest2594969-21munin-run21:55:003
253676599186,4cyclictest34-21ksoftirqd/223:00:212
2536765991811,5cyclictest436-20systemd-journal21:40:182
2536765991811,5cyclictest2624399-21grep23:15:182
2536765991810,6cyclictest2626202-21ntpq23:20:162
253676599180,17cyclictest0-21swapper/222:53:492
253676499184,12cyclictest3056-21PK-Backend20:58:531
253676499180,4cyclictest2573027-21sort20:50:161
2536759991816,1cyclictest0-21swapper/022:03:070
2536759991816,1cyclictest0-21swapper/021:52:380
2536759991813,3cyclictest2553046-21cron19:55:000
2536759991812,5cyclictest2625575-21cat23:20:110
2536759991811,6cyclictest2557374-21memory20:05:180
139099180,15rtkit-daemon2536753-21cyclictest20:45:390
253676999176,10cyclictest3056-21PK-Backend00:18:043
253676999173,13cyclictest1095-21runrttasks20:53:313
2536769991715,1cyclictest3056-21PK-Backend19:46:243
2536769991712,3cyclictest2578187-21memory21:05:183
2536769991710,5cyclictest2603406-21cat22:15:203
2536769991710,5cyclictest2603406-21cat22:15:203
2536765991713,3cyclictest2558249-21cron20:10:002
2536765991712,3cyclictest2545353-21timerandwakeup19:30:212
2536765991710,5cyclictest2634792-21tr23:45:162
253675999172,13cyclictest2595617-21grep21:55:130
2536759991715,1cyclictest0-21swapper/021:03:370
2536759991713,3cyclictest2542618-21/usr/sbin/munin19:25:140
253675999170,1cyclictest0-21swapper/021:24:530
253676999168,7cyclictest2646558-21apt00:20:123
253676999168,7cyclictest2646558-21apt00:20:113
2536769991610,4cyclictest2569885-21grep20:40:213
253676599168,6cyclictest2606224-21cat22:25:122
2536765991614,1cyclictest0-21swapper/221:14:322
2536765991612,3cyclictest2573069-21tail20:50:172
2536765991612,2cyclictest34-21ksoftirqd/223:25:212
2536765991612,2cyclictest34-21ksoftirqd/220:30:182
253676599160,7cyclictest2581668-21missed_timers21:15:182
253675999169,5cyclictest2588366-21sort21:35:100
253676999159,5cyclictest2636848-21sed23:50:223
2536769991513,1cyclictest3056-21PK-Backend23:42:563
2536765991511,2cyclictest34-21ksoftirqd/219:47:262
2536765991511,2cyclictest34-21ksoftirqd/200:25:172
2536759991511,3cyclictest2622209-21cstates23:10:110
253676999149,4cyclictest2579966-21sed21:10:173
2536765991410,3cyclictest3056-21PK-Backend19:55:032
2536759991412,1cyclictest0-21swapper/021:44:470
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional